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This report is prepared to two volumes* Volume I reports the Executive 
Summary and the findings of the research. V'olume n contains the ap|>endlces 
to the final re;x)rt. Itie ap|>erjdices list detailed di^cumentatlon which supports 
the research findings. This Includes specific materials .and procedures 
used In: a) the o|x'n and cIojihI forms of the knowledge teats, b) the full 
mission simulations I and c) the paper and |x*nctl tests. 
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EXECUTIVE SUMI^tARY 
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* A critical In-fUght event is a situation which Is unexpected, ucplaoned 

*r 

^ j and unanticipated « and Is perceived by the pilot In command to threaten the 

^ safety of the aircraft. The CIFE is one which requires pilot Judgment beyond 

" ^ routine decision maklzsg or a pre-programmed decision structure and v/here 

i the safety of the aircraft depends more on pilot cognitive processes than 

skilled motor performance. 

4 

Research Objectives 

The obje'^tlves of this research were to: 

1) Describe and define the scope of the critical In-fllgbt event with 
emphasis on pilot management of available resources. 

2) Develop detailed scenarios for both full mission simulation and 
paper and pencil (P/P) testing of pilot responses to ClFE’s. 

3) Develop statistical relationships among pilot characteristics and 
observed responses to CIFE’s. 

These objectives grew out of a concern with anomalies In reported accidents 
and Incidents In which some pilots or crews seemed better able to handle 
unusual In-flight events than others. For example, why did a professional 
crew piloting a Baltimore Colts 727 fail to recognize the symptoms of a 
j frozen pitot system and subsequently enter a fatal stall-spin maneuver? 

Contrast that event with the performance an airline pilot who used differential 
power to overcome a locked elevator p. ^blem on his three-engine aircraft. 

\\*hat characteristics of his training decision making strategy- permitted him 
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to develop a successful solution to tlie problem? Similar questions are 
raised by events such as a Cessna 206 pilot who experienced engine failure 
due to fuel exhaustion in one tank, and crashed the aircraft with the second 
tank nearly full of unused fuel. 

Project Developmen t 

The project began with an early concern for the dynamics of CIFE’s and 
broad attempts to Identify pertinent research issues. The final products 
were 1) a set of scenarios v/ith associated hardware and techniques for 
studying CEPE phenomena In a basic general aviation flight simulator; 

2) a set of paper and pencil scenarios and associated techniques for studying 
pilot diagnostic strategies and diversion decision making processes; 3) a set 
of testing Instruments designed to measure a pilot’s knowledge of aircraft 
subsystems and understanding of troubleshooting techniques; 4) a study 
relating cockpit crew procedural compliance with performance errors. 
By-products of this research included one M.S. design project, one M.S. 
thesis, and a Hi.D. dissertation^. Major milestones In the project develop- 
ment are summarized in Figure 1. 

Model 

A five-phase model of pilot CIFE response Is hypothesized on the basis 
of a) discussions with experts in industry and government and b) observations 
made about pilot performance in both simulator and paper/pencil scena'^ios. 


♦See Appendix B 


III 




o. 

r 


■> -o <J^ V' 

b , o c- 


^ 


FI C ,, 'yj - ^ *?* 


6 

[ O 




. P O 

0 




oO r 


c ° 
0 

.1 

o 

•f 


1 'I 


’{Iso 

• 

C 

,i ■ 


-'r 

i 




r 


»“ oo 
. ^■ 
i '■ 



Figure 1. 
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The five phases are: 

1) Detection 

2) Dia^osiG 

3) Option Generation 

4) Decision Making 

5) Execution 

Information seeking activities permeate all five phases of this process. The 
Inter-relatlonshlps and feedback among these phases are outlined in Figure 2. 

GAT-Scenarios 

A Singer GAT-1 flight trainer was modified to permit a variety of extra 
failure modes and to enhance data collection. Three scenarios were created 
to be tested in the GAT-1. These scenarios each Involved a critical in-flight 
event imbedded in what was otherw'ise a routine simulated IFR flight. Subjects 
went through a pre-flight planning phase involving a complete weather briefing, 
route planning, and filing of flight plan. Take-off, climb and enroute phase of 
each scenario began under normal IFR operating conditions. Real time ATC 
communications, including background conversations, w'ere used to enhance 
realism. Some 20-30 minutes into each simulated flight one of the following 
critical events was introduced: 
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HYPOTHESIZED PILOT CIFE RESPONSE 



EST. LIMITATIONS 


State of System 
Time Estimate 
Criticality Evaluation 



EXECUTE OPTION 


Unsatisfactory 




EVALUATE PROGRESS 


Monitor Systems 
Comoare States 
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:OM?LETE EXECUTION 


HYPOTHESIS: 5 Phase Model . Informat! 
Seeking Permeates All Phases. 

A. DETECTION: Items 1 thru 5 

B. DIAGNOSIS: Items 6 thru 8 

C. OPTION GENERATION: Item 9 

D. DECISION MAKING: Items 10 thru 

E. EXECUTION: Item 13 
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1) Fuel starvation on the active tank (as might be encountered 
because of a loose fuel cap). 

2) Partial power failure (as might be caused by a broken baffle In 
a muffler). 

3) Navald loss (as might be caused by failure of a single airborne 
receiver component). 

Subject performance was observed through one-way windows on the 
simulator and recorded by video tape, a 3-channel audio tape and written 
evaluations by the three experimenters present. These data were later used 
to measure ’’stick and rudder” skills and communications techniques as well 
as to map each pilot’s response to tlie critical In-fllght event. 

Twelve subjects w’ere selected for testing in the full mission GAT scenarios. 
Although all were IFR rated, they ranged in age from 20 to i>6 years old, in 
flight experience from 270 to 8800 hours and in certifica:n n from private pilot 
to ATP. Each subject was given two different forms of Icnowledge survey to 
complete and was thoroughly debriefed after his flight. 

A wide range of cockpit management styles and apparent skill levels were 
observed. Although it was difficult to quantify, ’’good performance” v;as 
easily recognized by the observers of the experiment. The elements of ’’good 
performance” included: 

1) professional use of tlie radio 

2) precise heading and altitude control prior to and during the CIFE 

3) constant awareness of the aircraft position along Its intended route 
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4) prompt, but not necessarily Inotant, response to the on-set of the 


CIFE (detection) 

5) systematic procedure for trouble shooting 

6) knowledge and use of available ATC resources 

7) diversion decisions which allowed for further potential uncertainties 
The sample was too small to indicate anything otlier than some initial 

hjHPOtheses concerning pilot performance in such a full-mission setting. 
Hov/ever, the following tendencies were noted: 

1) Cockpit management style varies widely among pilots. For 
example, some are extremely self-reliant, others want 
Immediate and extensive help from ATC while still others 
make the decision making process a joint effort with ATC. 

2) Good stick and rudder pilots seem to have excess capability 
and maintain good stick and rudder performance during and 
after the CIFE. More marginal stick and rudder pilots, on 
the other hand, show increased frequency and amplitude of 
heading and altitude excursions, and experience communication 
difficulties when faced with a CIFE. 

3) Pilots who score w'ell on the knowledge test instruments ter 5 
to perform well in problem diagnosis and decision making. 

From the observations of the experimenters and comments made by 
participating subjects, It appears that such a full mission simulation exercise, 
coupled with an appropriate knowledge survey and debriefing, could be a 
valuable tool for recurrent training of IFR pilots. 
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Paper and Pencil Scenarios 

I Paper and pencil (P/P) scenarios, and associated experimental techniques, 

|- were created to streamline the data collection and analysis for pilot responses 

to critical in-flight events. Although they lacked the high stress environment 

i 

I of the GAT- 1 experiments , these scenarios did yield useful data on pilot 

problem diagnosis and decision making skills and strategies. 

• • 

The paper and pencil scenarios have the following advantages over the 
I GAT-1 scenarios: 

, 1) Experimental conditions are more easily replicated between subjects. 

2) Data collection Is more easily accomplished. 

3) Diagnostic capabilities and decision making strategies can be more 
easily isolated. 

4) Tney are much cheaper. In terms of both time and money, which 
means that a much larger sample size of subjects can be run. 

The paper and pencil experiments were conducted In a workshop-type 
environment. A group of subjects, usually three or four, were seated in a 
conference room for a common briefing and Initial testing. Each subject was 
asked to complete a background questionnaire, which asked for data on his 
personal flying experience. Items such as age, ratings, total flying time, 
recency of experience and type of flying most often done were included. They 
then were given a 20-question knowledge survey (multiple-choice questions) 
designed to measure their knowledge of aircraft sub-systems and trouble- 
shooting skills. After the tests were completed, the group was given a complete 
briefing on the eauipment to be flown, the weather expected, and the airspace 
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in which they would be assumed to be operating^ for purposes of their CIFE 
scenarios. At that point the group was disbanded with each subject accompany- 
ing a single experimenter to a private room where the scenarios were administered. 

Two sets of scenarios were used on each subject. The first set consisted 
of four scenarios directed toward problem diagnosis. The second set 
involved two exercises designed to explore diversion-decision making strat- 
egies of pilots. (A diversion decision involves choosing an alternate airport 
when the intended destination airport Is unavailable due to a CIFE . At the 
completion of the paper and pencil experiments the subjects were Invited to 
tour the GAT-1 simulator used in the earlier study and to participate In an 
Informal debriefing. The entire process required about ninety minutes from 
beginning to end. 

For tliese tests, forty subjects were used. Almost all were current 
instrument-rated pilots with ages ranging from twenty to sixty-five years, with 
both civil and military baciqprounds , and embracing total flying experience from 
270 to 19,000 hours. As a group, these pilots were considered to be above 
average in experience. 

p/p Diagnostic Scenarios 

Four separate diagnostic problem situations were presented to each subject. 
These scenarios centered about problems presumed to be created by: 

1) an oil leak at the oil-pressure gauge line 

2) a vacuum pump failure 

3) a right magneto drive gear failure 

4) a frozen static port 
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After Instructions for the diagnosis scenarios were read to the subject, 
he was given an aircraft Instrument panel layout diagram and enroute chart 


for the first problem. The sf enario was then read, concluding v/lth a 
statement of a major symptom, e.g. , *Tfou smell hot engine oil. What 
would you do?'’. The subject was given a maximum of four minutes to seek 
information from the experimenter and conclude his diagnosis of the problem. 
He could ask for any information available from Instruments noted on his 
panel diagram, response to conti'ol inputs or external cues such as oil on 
the windshield or ice on the wings. The experimenter had a diagnosis infor- 
mation checklist from which be provided information in response to the 
subject's request. For example, if the subject asked for oil temperature, 
the experimenter would respond ’’normal*' if that was the entry on his check- 
list. As each piece of data was requested, its order was noted on the exper- 
imenter's checklist. If a diagnosis was not offered by the subject prior to 
the elapsed time ^four minutes) the subject was asked for his best estimate 
of the diagnosis at that time. At the completion of the allotted time the subject 
was asked to estimate the criticality (scale of 1 to 7) of the problem as he 
perceived it. Then he was given the correct diagnosis and was asked to 
re-estimate the criticality in the light of this perfect information. The same 
procedure was repeated for each of the four scenarios. 

Eight pieces of basic Information were extracted from each diagnosis 
summary sheet. These were; 
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1) Number of Inquiries. (An Inquiry represents a request for 
a single piece of information . ) 

2) Total tracks of Inquiries. (A track represe its a single coherent 
lino of questioning which may Involve several inquiries: for example, 
fuel pressure, fuel flow, fuel gauge status.) 

3) Unique tracks of inquiries. (A ^^ubject may suirt one track, 
abandon it, shift to a second track and then return to the first 
track. Althou^ three total tracks would bo noted, only two 
unique tracks exist. ) 

4) Correctness score- (A score of 1 to 5 was given which reflected 
hew close the subject's final diagnosis was to the ’^perfect” 
solution. ) 

5) Time to complete the diagnosis. 

6) Criticality estimate befere the correct diagnosis was revealed. 

This was a subjective-rating scale, 1-7. 

7) Criticality estimate after the correct diagnosis was revealed. 

8) Number of control Input inquiries. (A control inquiry involves 
movement of an aircraft control, e.g. , '\\Tiat happens If I 
advance the throttle?'*) 
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These primary data wvro thea used to create a number of compound 


i 

I pcrformaoco mcadures includit^ estimates of c(ficlczK:y and merit. 

^ **Efflclency** vas measured by the lime and number of Inquiries required 

to reach a diagnosis. Subjects who reached their diagnosis quickly (be it 
I right or wrong) and who made relatively few Inquiries received high efficiency 

^ scores. ’’Merit** was measured by multiplying correctness and efficiency 

scores on a given scenario. 

! 

P P PtH^islon Making Scenario 

The decision making phase of the paper and pencil experiments was 
divided into two pans, an Information seeking part and a rank ordering | 

of alternatives. The basic scenario used for both phases involved a hypo- 


vi ‘ 



O O 


o 

. 8 


thcnical night in a Cherokee Arrow from Bangor. Maine to Glens Falls. 

New York for a business meeting. Weather along the route and at the 
destination was marginal with rain, low ceilings and drizzle signifying 
instrument meteorological conditions (IMC). The scenario was read to 
the subject :is he was invited to follow* the progress of the hN'pothetlca! flight 
along an enroute chart. About midway along the rcute the aircraft encountered 
an alternator failure, the diagnosis for which was clearly defined for the 
subject. An upper limit on the length of time battery power alone would 
run die required electrical equipment was then ghen. This maximum time 
(exact time was uncertain) was loss than the time required to reach the 
primary desilnallon, thus forcing a diversion decision on the pilot. 
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For the information seckirg task tlie pilot waa supplied with a almpllfled 
enroute chart with sLxtccn airports indicated by letters alon^ his flight path. 

The subject was then given two minutes to ask for Information about any of 
those airports. For each airport questioned, there were six pieces of 
lnfori;.atlon the experimenter was prepared to provide: 

1) Bearing and distance from his present location. 

2) Ceiling at the airport. 

3) VM«»<bll(ty at the airport. 

4) Approach aids available. 

5) ATC services available. 

G) Terrain surrounding the airport. 

The experimenter provided the pilot with each piece of Information requested and 
the experimenter recorded the sequence in which it was requested. The pilot 
continued to request Information until he had selected an airport (or until 
forced to select at the end of two minutes) and revealed his choice to the 
experimenter. 

For the ranking of alternatives phase, the pilot was asked to rank each 
of sixteen alternative airports. Ke was provided with ATC facilities, ceiling 
and visibility, time to be reached and approach aids information on all airports. 
The airports were to be ranked from "most preferable" to 'least preferable" 
given his problem situation. No time limit was Imposed for this task. In 
order to assess his risk-taking tendencies, the experimenter posed a series 
of questions for the pilot to consider after he had obtained his ranking. The 
questions asked how far he would go down his list of ranked airports to find 
one with maintenance facilities to repair his airplane. 
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The data from the raaking task were used to detormlRe tho relative worth 
structure. The coefficients, or weights, for tl\e variables ATC, weather, 
time, and approach were obtained by regression analysis according to the 
techniques of conjoint measurement. The ran^e of values for the coefficients 
was 0.250 to 4.000. The relative worth coefficients were later used to 
determine if any relationship existed with pilot backgrouiid variables, results 
of the knowledge survey, diagnostic ablHt>', and search pattern exhibited In 
tlic Information seeking task. 


P/P Scenario CfOnerallzatlons 

For purposes of analysis the closed form (multiple choice) knowledge survey 
was considered to be part of the P/P experiments. This knowledge survey 
focused on aircraft subsystems and trouble shooting In three major areas: 

1) engine and fuel systems, 2) electrical systems and cockpit Instrumentation, 
and 3) weather and IFR operations. 

A series of Spearman Rank Correlation studies, stepwise regression 
analyses and t-tests were p>erformed on the combination of pilot background 
variables, knowledge survey results, diagnostic scenario performance and 
decision making measures. Among the observations made from these analyses 
are the following: 

1) There Is no correlation between knowledge score and total 
night hours. 

2) Knowledge score is correlated with pilot ratings held. 

3) Pilots good in one section of the knowledge survey tend to be 
good in all sections. 
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4) Diagnostic performance Is highly correlated with knowledge score?. 
I 5) Knowledge Is Inversely related to total diagnostic Inquiries, 

knowledgeable pilots reach conclusions (right or wrong) 
more rapidly than others. 

I 6) Total diagnostic Inquiries is inversely related to correctness, 

. This suggests that undirected experimentation Is poor diagnosis 

* style. 

7) Total diagnosis correctness score is correlated with efficiency. 

8) Civil trained pilots place a higher wortli on ATC service in 
diversion decisions than do military pilots. 

9) Private pilots place a higher worth on weather factors in 
diversion decisions than do commercial and ATP rated pilots. 

10) ATP rated pilots place high worth on time in diversion decisions. 

11) Pilots with good dlagnaostic scores place less wel^t on approach 
aids in diversion decisions. 

12) Pilots with good diagnostic scores place more weight on time In 
diversion decisions. 

13) The pilots with good diagnostic performance w'ere characterized 
as knowledgeable about aircraft systems, employed few tracks 

to get at an answer, used few inquiries per track, and emphasized 
time in their destination diversion decision. They were not 
differentiated by flight hours, ratings, training, or t>pe of flying. 
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Procedural Compliance 

j In support of the general research objectives, Schofield Investigated 

j airline cockpit crew operations. For his dissertation he used data generated 

^ in an experiment conducted in 1976 by I)r. H. P. Ruffell Smltli under the 

auspices of the NASA-Ames Research Center*, Rxxffell Smith used a full 
mission simulation scenario of a Boeing 747 fli^^t to study crew errors 
generated during high workload segments of the simulated flight. Schofield 
used the same data to study routine tasks of flight operations during low work- 
load segments of that flight. He vras concerned with: 

1) Quantifying routine procedures. 

2) Analyzing observed crew errors to identify which particular crew 
members were the primary causes of such errors. 

3) Comparing measures of procedural compliance and operator error. 

Schofield identified nineteen separate words and phrases associated with 

aircrew operations which had procedural connotation. Using that Mst as the 
basis for definition he enumerated 97 normal operating procedures which 
could be Identified as standard cockpit activities In a 75-mlnute flight. This 
list did not include any abnormal, alternate. Irregular or emergency pro- 
cedures. 

Twenty-one crew coordination procedures were separated from the total 
list for further study. This group was emphasized because those procedures 
captured the essential ingredients of group leadership, crew management, 

♦See Appendix B 

r 


XVtl 



and behavioral conformity. Schofield sought to examine :elatlonshlps between 
meticulous compliance with coordination procedures and the crew errors 
noted by Ruffle Smith. 

Schofield selected ten runs, which had the same set of observers and 
usable audio data throughout, for detailed procedural analysis. The 21 crew 
coordination procedures were further subdivided into check lists, call outs, 
configuration changes and transfers with each of the ten crews evaluated In 
each subdivision. 

The prescribed command-announcement-challenge sequence for check- 
list procedures was fully executed in only five of fifty opportunities, when the 
crew members involved were pilot and co-pllot. W^en the flight engineer 
was Involved, fifteen of thirty opportunities were fully executed. Schofield 
hvipothesized that crew coordination might be Improved by making the flight 
engineer the challenger of all checklists. 

One hundred seventy opportunities, among the ten crew^s, to execute 
callout procedures were noted. Thirty eight procedural errors were 
Identified, half of which were errors in altitude callouts during climb or 
descent. 

The 104 observed configuration changes, e.g., gear and flap extensions, 
were well executed In terms of established oral procedures. Verbal 
indicators of transfer of EGT monitors were also given with few omissions. 
However, the optional transfer of control procedure was seldom observed 
even though opportunities existed to use It. 

xvlll 




• o o 


c.' o 


o 


c 


Oc 





o 


I 

I 

I 

Schofield used stepwise multiple regression techniques to Identify the 
best models relating the Independent (procedura./ variables to each of the 
dependent (error) variables in turn. He found that dependent variables 
. which reflect errors by the flying pilot, by the captain, and by the two 

pilots collectively all have highly significant regression models in which 
pilot flying checklist commands and non-flying pilot callouts are the common 

) 

Independent variables. 

j The Schofield study of procedural compliance by aircrews who participated 

j In the Ruffell Smith experiment suggests the following observations: 

t 

1) Crew members face an impossible challenge in attempting to 
mentally catalog all of the standard operating procedures (SOP) 
published for them. 

2) Routine non-compliance with ar assortment of SOP*s has been 
documented. 

3) Human redundancy by Itself does not erradlcate personnel 
errors. 

4) A statistical link appears to exist between operator errors 
and procedural compliance. 
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1 . INTRODUCTrON 

Departure: 75Y what Is your altitude? 

N1G75Y: Columbus Departure Control, what do you show our altitude on 
our encoding altimeter? 

Departure: I show you at 500 feet. That’s why I asked you. 

N1675Y: It shows 1000. 

Departure: OK. Stop altitude squawk. I show you at 400 feet now. 

Obviously it’s not working right. 

N1675Y: 75Y we’re having problems with airspeed and ever\*thing here — 
Wliat do you show our airspeed ? 

Departure: OK. 75Y do you want vectors back into the airport? 

N1675Y: Yeah! Lot’s do that. 

The above brief excerpt of an actual communication between ATC and a pilot 
experiencing in-flight problems in IFR conditions Uplfies a persistent dilemma 
in aviation. ^ We do not (nor does the air traffic controller) understand the true 
nature of this pilot’s problem. How long has he experienced airspeed and altitude 
problems? Is It a matter of structural Ice, mechanical failure or pilot error? 
If an emergency Is to be declared, what does the fact of declaring an emergency 
mean to the air traffic controller? What does it mean to the pilot? (A confession 
of Incompetence — an Invitation to loss of license? Is there a need to specify 
Intentions? Can the pilot pro\dde intentions If he Is unaware of the options 
open to him? How can we avoid those situations In which the pilot relinquishes 
command to someone on the ground? 


^These excerpts from a communications tap? are verbatim. Only the air- 
craft identification number has been changed. 
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The assessment o£ the criticality of the situation demands more Information, 
such as the pilot's capability, his training level, his experience with In- flight 
problems, weather, location, terrain, altitude, etc. Yet this situation Is but 
an example of many such events that occur each year In our national aviation 
system. 

Each year air traffic control provides several thousand assists to pilots. 

In 1970, of the 4, 187 assists, 53% involved lost pilots, but 25% Involved 
fuel problems, navigational failures, and mechanical problems. How many 
problems went unannounced and resulted la tragic consequences for lack of 
pilot understanding of how to cope with in-flight problems ? How' many 
emergencies were declared which could have been avoided and reduced dis- 
ruption in air traffic control systems ? 

Discussions with pilots of various experience levels and ratings reveal 
little agreement as to when to declare an emergency and the operational and 
legal consequences of such a declaration. There are instances wherein pilots 
have risked and lost their lives and those of the passengers to avoid possible 
suspension of license as a consequence of declaring an emergency when they 
believed they had violated a regulation. (See NTSB-AAR-71-1). Do the 
perceptions of the air traffic control personnel differ from that of pilots In 
this regard? Most importantly, can pilots be trained to handle In-fllght 
problems, provide early assessment and intelligent response to the situation? 
WTiat should a pilot do if: 

a) strange noises occur? 


b) the door opens In flight? 


CCo 
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c) the ammeter shows a discharge? 


! 

I 

] 

I 


d) the radios fail ? 

e) smoke appears in the cockpit? 

f ) he cannot determine his location in his flight progress ? 

g) the weather closes in on him? 

Some empirical evidence of pilot perception of threatening experiences is \ 

shown in Tables 1 and 2. These data suggest that a pilot’s ratings, type of 
operation f and Implied skill level, all serve to alter his perception of critical 
in-flight events. 

What should the ’’system'* be able to do to assist the pilot in properly 
assessing his real (or perceived) problem? No simple answer exists for 
these situations nor does past research appear to address these issues. It 
is hard to imagine the extent of myths and misconceptions about critical in- 
flight events. Critical events lead to air traffic control disruption, panic, 
accidents, and perhaps firm resolutions by pilots never to fly again. 

In the NASA Aviation Safety Reporting System, 1,497 incidents were 
submitted in the period of July 15 to October 15, 1976. Of these 3% involved 
aircraft structure and subsystem factors and about 9^ navigation and communi- 
cation situations. If one assumes that the Reporting System captures only a 
portion of the total incidents occurring in the system, this statistic also 
suggests there may be tliousands of critical In-tlight events each year. 

WTiile one objective of this research is to describe and define the scope of 
the critical in-flight event (CIFE), a definition or set of qualifiers for the 
purposes of this report is set forth below. 
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I-l. Reported Most Uncomfortable or Threatening EKperlenco During An IFR Flight 
In Actual IFR Conditions (Reproduced from Study to Determine the Operational 
Profile and Mission of the Certificated Instrument Rated Private and Commercial 
Pilot, FAA-RD-70-51 Juiy 1970, p. 125) 


# 

Experience 

General 

Number 

Avn IFR 

% Of 
Total* 

Number 

Total 
% of 
Total** 


(2) 

(3) 

(4) 

(5) 

Structural icing 

212 

25?; 

331 

is;; 

Thunderstorms 

91 

12 

262 

15 

Turbulence 

41 ■ 

6 

113 

6 

Conununlcaticna loss 

36 

5 

82 

5 

Equipment malfunction 

30 

5 

62 

5 . 

Engine failure 

20 

4 

67 

4 

Feoling behind a situation 

25 

3 

51 

3 

Deteriorating weather 

20 

3 

77 

4 

Approach to minimums 

20 

3 

• 59 

3 

Spatial disorientation 

19 

3 

73 

4 

Loss of navigation equipment 

IS 

2 

37 

2 

Kear midair and/or unknown traffic 

10 

2 

69 

5 

Loss of primary flight Instrumenta 

-13 

2 

35 

2 

Communications and navigation loss 

13 

2 • 

22 

1 

Unclassified 

93 

13 


15 

None or no response 

52 

7 

130. 

7 


* Total « 739 

** Total » 1767 
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Table 1-2. Most Threatening Experiences Reported by Private 
PQots (Reproduced from Study to Determine the 
Flight Profile and Mission of the Ccxlificated 


k . 

Private PUot, FAA-DS- 

•63-15, July 1SG8, pp. 

81-82.) 



Reports 


i 

Tnrentejiina Fj^jtrience 

Humber % of 

Total 1/ 

1 

(1) 

(2) 

(3) 

1 

Lew visibility 

333 

28?S 


CroGEwind 

290 

24 

r • 
) 

Low ceilLng 

277 

23 


Malfunctions 

254 

21 

i 

Landings 

247 

21 

i 

t 

High winds 

229 

19 

, 

In fog or clouds 

209 

13 

1 

Near air collision 

198 

17 


Lost 

148 

12 

r 

1 

Short field 

147 

12 

1 

'Fuel supply 

139 

12 

r 

! 

Engine operations 

122 

10 

Forced laxiding 

117 

10 


Takeoffs 

111 

9 

! 

Unimproved airport 

5S 

3 

f 

i 

Mud or snow 

96 

8 

Darkness 

92 

8 


Weight or loading 

83 

7 


Infrequent piloting 

87 

7 

f 

Trees or wires 

71 

6 


Use of radio 

64 

5 

i 

Soft field and high grass 

64 . 

5 


Different type aircraft 

42 

4 

1 

Preflight operation 

38 

3 

1 

Unavailable preflight information 

35 

3 
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Table 1-2 (Continued) 


Heportg 


Threatenincr Fxoarience 

IlambGr 

% of Total jy 

(1) 

(2) 

(3) 

Impropex airspeed 

30 

3 

Stalls or recoveries 

23 

2 

Low altitude maneuvering 

25 

2 

Uninformed 

2? 

2 

Steep turns 

21 

2 

Flaps 

20 

2 

Handling of aircraft 

19 

2 

Holding altitude 

18 

2 

Check list 

14 

1 

Slow speed flight 

13 

1 

Flight materials (maps, etc.) 

13 

1 

Pontoons or skis 

11 

1 

Slips 

10 

1 

Other 

136 

11 • 


u Total =» 1,192 
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A critical in-flight event Is a situation which is unexpected, 
unplanned, and unanticipated, and is perceived by the pilot 
in command to threaten the safety of the aircraft. The CIFE 
is one which requires pilot judgment beyond routine decision 
making or pre-programmed decision structure. It may or 
may not Involve communication with ATC. The CETE assumes 
alternative courses of action are open to the pilot and some 
finite period of time is available to the pilot to malce an assess- 
ment of the situation, enumerate options and make a decision. 

The safety of the aircraft depends more on pilot cognitive pro- 
cesses than skilled motor performance. 

For purposes of this research, emphasis was placed on IFR rated pilots who 
have sufficient exp^erlence to utilize tlie ATC system when available. 

Many examples of the above description can be put forth. The following 
Illustrates a few of these. 

a) failure of navigational equipment, 

b) failure of electrical systems, 

c) failure of hydraulic systems, 

d) fuel management problems, 

e) flights into unexpected weather, 

f) unforecast icing conditions, 

g) engine failure (single and multiengine aircraft), and 

h) partial pilot incapacitation. 

This research was directed towards an understanding of: 

a) the nature of critical in-flight events (CIFE), their causes, and 
how they develop over time; 

b) how pilots of different backgrounds might assess and respond to 
such instances; 
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c) the psychological stress of In-flight events, appropriate coping 
processes, and the modeling of such processes; 

d) the interaction that exists between air traffic controllers and 
pilots during CIFE*s; and 

e) how adequate countermeasures can be developed from the above 
to minimize the frequ.ency and consequences of CIFE's. 

An explicit description of research objectives and discussion of \he scope 

of the project are presented in the next section. 
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n. RESEARCH OBJECTI\"EG AND PROJECT SCOPE 


The general objectives of the research were: 

1. To describe and define the scope of the critical In-flight event with 
emphasis on characterizing 

a) event development, 

b) event detection, 

c) event assessment, 

d) pilot information requirements, sources, acquisition, and 
interpretation, 

e) pilot response options, 

f ) pilot decision processes, 

g) decision implementation, and 

h) event outcome. 

2. To develop detailed scenarios from (1) above for use in 

a) simulators as well as paper and pencil testing for developing 
relationships between pilot performance and background information, 
and 

b) an analysis of pilot reaction, decision, and feedback processes. 

The scenarios are viewed as data generating devices for pilot options. 
More specific thrusts of this research, related to the gcieral objectives 
above, were developed on the basis of initial research findings and research 
capabilities. These involved: 

a) emphasis on general aviation IFR pilots in single engine aircraft 


b) emphasis on the descriptive character of pilot response to 
critical in-flight events 

c) use of full mission simulation 

d) use of paper and pencil scenarios to study pilot problem diagnostic 
capabilities and destination-diversion decision processes 

e) exploration of the relationship between procedural compliance 
and flight crew errors using the Ruffell-Smith simulation data. 

The following chapters place major emphasis on: 

1) background activities leading to problem conceptualization 
(Chapter ni) 

2) development of knowledge tests on system anomalies (Chapter IV) 

3) full mission simulation (Chapter V) 

4) paper and pencil scenario tests (Chapter \T), and 

5) analysis of the Ruffell-Smith data for procedural compliance 
(Chapter VII) 
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m. BACKGRCXJND 

A. Literature Review 

Initial project activities centered around the development and Implementa- 
tion of a comprehensive iltcrcmrc aearch. Eecsuse the objectives of the 
project v/ere rather broad ranged and cross-disciplinary, this search Involved 
a number of topic areas. After an extensive review of search materials 
available a master ’1st of key words to be used In all literature s»ar;hes 
was developed. This list 'vas used for all searches with the exception of 
pscyhology abstracts which used a controlled vocabulary. This controlled 
vocabulary can be found In The Searches of Psj'chological Index Terms, pub- 
lished by the American Psychological Association. The following sources were 
examined: 

a) The Ohio State University Mechanized Information Center (OSU-MIC) 

b) Psychology Abstracts 

c) FAA Accident Reports 

d) Transportation Research Information System (THIS) 

e) National Technical Information Service (NTTS) 

f ) Department of Defense sources (see Appendix A) 

g) Aviation Press Publications, e.g. , Flying Magazine, Business and 
Commercial Aviation, etc. 

h) National Transportation Safety Board (NTSB) Accident Reports 
While the literature was replete with "never again" stories, surprisingly 

few documents addressed pilot response to critical In-fllght events In 
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sufficient detail to permit pilot response modelling. Indeed. little 
statistical evidence was available on the relative frequency of various types 
of incidents. 

Appendix B is an annotated bibliography of some of the literature examined. 
In addition* the dissertation by Schofield and the thesis by Flathers detail 
further background sources in this area. 

B. Results of Interviews With Interested Agencies 

At the outset of this project the principal investigators met with several 
organizations which had both a vital interest In the problem and expertise in 
pilot behavior. The National Transportation Safety Board iNTSB), The 
Aircraft Ouners and Pilots Association (AOPA), Mitre Corpi>ratlon, Airline 
Pilots Association tALPA), Air Transport Association (ATA), National 
Aeronautics and Space Administration (NASA), Federal Aviation Administration 
(FAA), Air Force Office of Scientific Research (AFOSR), and United Airlines 
(UAL) wore all visited to provide consultation with their staffs on their t>er- 
coptions of the CIFE and to secure whatever data bases were available to 
document the extent and nature of CIFE and any data on related pilot response. 

V agencies also suggested other resources for this problem area - either 
y. ‘^hed reports, research In progress or names of individuals who could 
provide Insight into the CIFE problem. Trip summaries and contacts are 
outlined In Appendices B and C. 

In general, all agencies reported a great Interest In the problem and were 
willing to help within their capacities but admitted that the CIFE was largely 
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an unresearched Issue. No data bases on pilot response to CIFE*j were 

available. There were, to be sure, many shared experiences, Individual 1 

examples of CIFE*s from FA A and NTSB files and unique perceptions from 

those interviewed. For general aviation (GA) ClFEs there were little or no 

data available from NTSB, FAA or NASA/ASRS files. From discussions 

with these agencies and among the research staff, several hypotheses or 

constructs were proposed about the C!FE process such as 

1) response latency theory 

2) social interaction In the cockpit 

3) cognitive structuring 

4) pilot w'orkload 

5) detection of vs. response to CIFE 

G) appraisal of CIFE*s 

7) single channel limitation of the pilot 

8) lack of standard work habits 

9) lack of real world elements in training and testing of pilots 

C. Results of the NASA-ASRS Search 

Early in the research, the project team asked NASA-Ames to perform 
a search of its ASRS data file. Using key words consistent with their data 
base structure, e.g., emergency, pilot decision making, etc., some two 
dozen narratives were developed and examined. In general, little value to 
the project resulted from this search principally because of lack of detail 
about how the problem developed, how it was diagnosed, what alternatives 
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were considered and other relevant details, e.g. , weather and alternate 
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airports available. Because of the NASA policy on anonymity of the reporter It was 
- impossible to trace back an Incident to get more Information. What would 

have been necessary would have been to have the analyst follow up Immediately 

« t 

on first contact using a supplementary data sheet. Because the tlirust of the 
research was directed towards the GA pilot itnd die fact that the majority of 
Incidents reported were air carriers, it was decided not to pursue Uie 
ASRS data file further. 

♦ 

D. An Initial Conceptual Model 

I 

As a result of background information, discussions with experts and a 
graduate seminar directed to the various facets of the problem, a preliminary 
model of this process evolved. This Is shown In Figure III-l. This conceptual- . . 

izatlon depicts several key asp<'cts of the problem - the detection phase, 
information seeking strategies, workload, use of resources, and pilot 
stress, oeclslon styles and value systems making up his decision making. 

U!ti/ lately, pilot response was focused upon: 
i . detection of the problem 
; diagnosis of the cause from the symptoms 

3 ) generation of viable options 

4) decision making both in terms of problem resolution and 1 

destination diversion 

5) execution of the 'oclslon 

• Throughout all fr*e phases. ^ llct Information seeking strategies were studied. 
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A MODEL OF RESPONSE TO A CIFE 
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E. Research Strate;^ 


I » 

Figure III-2 depicts the overall strategy undertaken In this research. 

Above the dotted line are the Initial background efforts. These Included the 
aforementioned visitations, literature review, ASRS search and graduate 
seminar. Prior to the formal research initiation, a graduate project by 
Fox, ’’Critical In-Flight Responses", indicated the potential value of using 
paper and pencil scenarios to study pilot decision making. At the same time, 
as part of another Industrial and Systems Engineering iiSE) course, USAF 
pilots were surveyed to arrive at candidate scenarios for future simulation 
or paper and pencil testing. Both of these exercises provided encouraging 
results. 

Below the dashed line are the four major fronts of the project: 

1) the development of knowledge tests 

2) fall ir.ission GAT simulations 

3) paper and pencil scenario testing 

4) the relation of procedural compliance to errors in the Ruffell 
Smith simulation study 

Each of these major fronts is discussed in turn In the chapters which follow. 

A comment on the paper and pencil scenario workshop is in order. This 
was a mechanism to bring pilots together for a general briefing prior to 
their testing on paper and pencil scenarios. These tests were then conducted 
with Individual experimenters. Hence, about ten small group w'orkshops were 
held on different days. 
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IV, EVALUATION OF SUB^JECT KNOWLEDGE 
OF AiaCHAFT SYSTEM ANQ?1ALUI:S 

The evaluation of subject knowledge of aircraft systems and the IFR 
operating environment was an Important element in understanding the ways 
In which pilots respond to CIFE’s. The evolution of the final test instrument 
involved a series of pre-tests, development of open form questions and finally 
a clo^^ed-form, multiple-choice questionnaire. Contrary to the usual airman 
certification exam question format, the bulk of the Items selected here 
emphasized aircraft subsystem operation and trouble-shooting. 

A . Open Form Survey 

A knowledge survey, or Inventory, was developed to determine the level 
of a pilot’s knowledge of aircraft systems and the IFR operating environment. 

An open-form survey was administered to pilots who were participants in the 
GAT runs. Later a closed- form version was administered to all subjects In 
the paper and pencil scenarios. The results of the surveys were compared to 
various measures of pilot performance In the simulations to Isolate relation- 
ships between pilot Imowledge level and measureable aspects of piloting skill. 
The development of the survey followed a three stage process which Included 
1) Item (question selection and pre-test, 2) construction and test of an open-form 
survey, and 3) construction and test of a closed-form (multiple choice) exam. 

The items for the survey were constructed from Information In training 
texts, government publications, aircraft operating manuals, and other 
readily available publications. Practice quizzes and examina- 
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tlons commonly used In the certification process placed much emphasis on 
such areas as regulations, weather, navigation, and weight and balance* 
Relatively few Items on aircraft subsystem operation or trouble-shooting 
could be found. This Is due, in part, to the fact that a modern single engine 
aircraft may have over 25 independent systems, aud some of these systems 
may be engineered differently by the various manufacturers. For example, 
while the pitot-static system and g 3 TO- Instrument system designs offered 
by aircraft manufacturers are fairly uniform, other systems such as fuel 
metering and feed devices are often vastly different. Special care was taken 
to ensure that items selected for the knowledge survey were representative 
of the types of systems pilots could be reasonably expected to encounter In 
their flying careers. 

A total of over 60 items were collected and pre-tested on a small group of 
pilots. Included in this prototype survey were areas such as fuel systems, 
electrical systems, engine systems and operations, cockpit instrumentation, 
weather and the flight environment, and general IFR procedures. The pre-test 
survey items were presented in the form of open-ended questions to which the 
pre-test subjects responded with short, written ariswers. Four types of 
questions were posed. The first tj^pe was a simple, straightforward question 
in which the pilot was asked to define or explain something. In the second type 
of question, given certain symptoms In terms of instrument indications, noises, 
visual inspection, and the like, the pilot was asked to identify the most likely 
cause of tliose symptoms (Symptom-Cause, or S-C). In the third type of 
question, the pilot was given a specific condition and was asked to identify 
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symptoms that would most likely arise from that condition (Cause-Symptom, or 
C-S) . In the fourth und final type of question, the pilot was asked what 
corrective action should be taken If a certain condition was known to exist 
(Cause-Correction, or C-C). 

The correctness of the responses of the pre-test subjects was not as 
important as the ease with which the subjects understood and responded to 
the questions. The experience gained in the pre-test was very helpful In 
determining which questions were not useful and should be eliminated. It 
was also helpful In determining the way in which the remaining questions 
should be streamlined to improve claritj^ All of the improvements suggested 
by the pre-test were made and the end result was the refined, open- form 
knowledge survey which was used in conjunction with the GAT simulation 
studies. 

The open form survey, contained in Appendix D, consisted of 58 questions 
which called for short, written answers. Thirty questions v/ere of the 
straightforward 11 were of the C-S type, 9 were of the S-C type, and 

8 were of the C-C type. The open-form survey measured overall pilot know- 
ledge, as well as knowledge In the six areas listed In Appendix D. Scoring 
of the survey was performed with the aid of the answer key also provided In 
Appendbc D. Partial credit was awarded for answ^ers which came close to 
those given in the answer key. 
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For evaluation of the paper and pencil simulation tests, pilot knowledge 
about systems was needed. This was accomplished through the use of the 
closed-form knowledge survey given in Appendix E. The survey consisted 
of 20 multiple choice questions. Nine of the questions were of the straight- 
forward type, 6 were of the C-C type, and 5 were of the C-S type. All of the 
questions came from the open form survey and were selected on the basis of 
their ability to discriminate between good pilots and marginal pilots. Some of 


the incorrect responses offered by subjects in the open form survey were used 
as ”dummy” alternatives in the closed, multiple-choice form. 

Under the multiple choice format all subjectivity in scoring was removed, 
and the time spent administering and scoring was greatly reduced. The answer 
key is given in Appendix E. Scores were provided for the three pilot knowledge 
subscore areas also given in Appendix E, as well as for overall pilot knowledge. 
Results of the closed form survey are discussed in Chapter II covering the 
paper and pencil simulations. 


! 
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B. General Results of the Closed-Form Knowledge Survey 

The closed form knowledge survey was administered to forty pilot-subjects, 
thirty of whom were also participants in the pilot decision-making workshop. 

The mean total score for the forty subjects w'as 12.4 with total scores ranging 
from five to seventeen. The maximum possible score was twenty. 

Statistical tests were performed to determine if any relationships existed 
between knowledge survey scores and pilot background variables. The Spearman 
rank coefficient was used as a measure of correlation throughout. A summary 
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of the correlation coefficients between total knowledge survey score and 
pilot background variables appears in Table IV- 1. 



Table IV- 1 

Total Knowledge Score Spearman Correlation Coefficients 
and Observed Level of Significance (In Parantheses) 


Total Flight Hours 

.131 

(.42) 

IFR Hours 

.002 

(.99) 

Single Engine Hoiirs 

.467 

(.002)* 

Rating 

.430 

(.006)* 


♦means significant correlations at p <. .05 level 


As seen In Table IV- 1, almost no correlation exists between total taowledge 
survey score and total hours or IFR hours. These lack of relationships suggest 
that accruing general flight experience or IFR flight experience does not 
guarantee knowledge will also increase. One possible explanation for this 
observation, however, is that, as pilots accrue more and more flight time, 
they tend to advance to more sophisticated aircraft with sharply different 
operational characteristics. The knowledge survey was aimed at the single 
pilot IFR operations common in light aircraft. These two relationships may 
not be as strong, then, because pilots v/lth more flight experience may have 
moved out of the scope of the knowledge survey. 

Substantial positive correlations are seen, in Table IV- 1, bet^*een total 
knowledge survey score, and single engine hours and rating. These two 
relationships lend support to the knowledge survey*s validity as a general 
tool to measure knowledge of single-pilot IFR operations and aircraft systems. 
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One would expect increased exposure to single engine operations (more single 
engine hours) would also increase a pilot’s knowledge of single engine operations 
(which was the focal point of the knowledge survey). Additionally, as one’s 
tested level of competence Increased, knowledge should also Increase. 

The knowledge survey was broken into three subcategories: Engine and 
Fuel Systems, Electrical Systems and Coclqpit Instrumentation, and Weather 
and IFR operations. They were named Category I (CATSCR 1), Category n 
(CATSCR 2), and Category HI (CATSCR 3), and contain 7, 7, and 6 Items, 
respectively. The means and range of scores for each category for the 
(cTty subjects are given in Table IV-2. 

Table IV-2 

Klean Scores and Ran^e of 
Scores for Categories I, II, and ITT 

mean % range 
(%of max 

mean possible) low high 

Category I (maximum possible = 7) 4.82 68.9 1 7 

Category n (maximum possible = 7) 3.750 53.6 2 7 

Category III (maximum possible = 6) 3.850 64.2 0 6 

The SI me correlation tests were applied to these scores as w*ere per- 

formed on tlie total knowledge survey score. A summary of the correlation 
coefficients betw'een the three category scores and pilot background variables 
is given in Table IV-3. 
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Table IV- 3 

Correlation Coefficients and Ob se rved Level of Stfmificance 
Category II, and m Scores » and 


Back^ound Variables (Levels of SiCTiticance In Parantheses) 



Total Fll^t 

IFR 

SE 



Hours 

Hours 

Hours 

Rating 

Category I Score 

.197 (.22) 

.090 (.58) 

.474 (.002)* 

.273 (.09)* 

Category II Score 

-.115 (.48) 

.194 (.23) 

.184 (.26) 

. 195 (.23) 

Category ni Score 

.120 (.4G) 

-.001 (.99) 

.S76 (.017)* 

.376 (.017)* 


♦indicates significant relationships at p < . 10 level 

As evident in this table, Category I score (engine and fuel systems) is 
positively correlated with single engine hours and ratings, whereas no 
significant correlation exists between Category I scores and total flight time 
or IFR flight time. These results may be due, again, to the fact that the 
knowledge survey was aimed toward single pilot IFR operations. There are 
sharp differences in powerplants between the s 'ph*sticated airplanes exper- 
ienced pilots are more likely to fly and the simpler, lighter crafts flown in 
single pilot operations. This is particularly tioie when one considers the fact 
that higher performance airplanes are often powered by turbojets or turbo- 
propellers. 

Category m score (Weather and IFR operations) is positively correlated, 
again, with single engine hours and rating, and uncorrelated with total flight 
time and IFR hours. No correlations were found In any case involving Category n 
(Electrical Systems and Cockpit Instrumentation). 
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A summary of tndlviduatl subject kr.owledge survey performance, including 
results for each of the three subscores is found in the Master Data Table, 
Table VI-4 In Chapter VI. 
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V, CAT FULL MISSION SCENARIO (FMS) RUNS 

» 

t 

Prior to the dovolopmcat of the final paper arid pencil e?cperlmenl3, a 
series of full mission simulation (FMS) experiments were performed. These 
experiments, which are described below, provided background for designing 
paper and pencil scenarios and a benchmark which such scenarios could be 
matched for a rudimentary cost/boneflts evaluation. 

A . Purpose 

A Singer GAT-1 flight trainer was reconfigured to serve as a flight simu- 
lator for use in **LOFT** type scenarios. Thvse scenarios each Involved u 
critical In-fllght event Imbedded within an otherwise iiormul simulated IFR 
flight mission. The purpose was to gain an understanding of: 

a) how pilots of different backgrounds assess and respond to 
such Instances; 

b) the psychological stress of In- flight events, appropriate coping 
processes, and the modeling of such processes; and 

c) the Interaction that exists between air traffic controllers and pilots 
during CIFE*s. 

In keeping with the full mission scenario approach, each subject went 
through a pre-flight planning phase Involving a complete weather briefing, 
rcxito planning, and filing of flight plan. Take-off, climb and enroute phases 
of each scenario began under normal IFR operating conditions. Real time 
ATC communications, including background conversation were used to 
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enhance realism. A critical event v;as Introduced some twenty to thirty 
I minutes Into each simulated flight. 

I The conduct of an FMS la outlined in the paragraphs which follow. Com- 

I 

plcte operating Instructions and detailed supporting material are contained 
In the Master Notebook for GAT Scenarios, a copy of which Is available in 
thl V projevl’s file at NASA- Ames. 

Be E.xperlmental Equipment 

The primary piece of equipment used In the full mission simulation studies 
Is a Singer General Aviation Trainer (GAT-1) on a motion base. Three degrees 
of freedom, roll, pitch, and yaw, are provided by the motion base. This 
machine simulates, both in design and performance, a t 3 *plcal single engine, 
carbureted, fixed pitch prop, fixed gear aircraft, 'fhe avionics equipment 
Includes dual navigation and communication radios, dual VOR indicators (one 
with glideslopc), an automatic direction finder, an audio control panel, and a 
throe-light marker beacon receiver. 

Modifications have been made to the standard GAT cockpit. A trans- 
ponder and a digital clock have been added to the instrument panel. A fuel 
selector switch has been Installed to the left of the pilot's seat. A lapel 
microphone has been added to pick up the pilot's communications and cockpit 
sounds. Two floodlights and a closed circuit television camera have been 
mounted over the pilot's right shoulder to view the Instrument panel. 

External modifications have also been made to facilitate the experiment. 

The windows of the GAT have been covered with a one-way reflective film 
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(Scotchtlnt) so that the pilot can be observed during the flight without his 
knowledge. A display ior chewing which fuel tank is active has been Installed, 
as well as external controls for the ammeter, panel light intensity, and for 
power (rpm) reduction. These are all new additions to the standard GAT 
hardware. 

The experimenter has the capability to control tiie operational status of 
some of the GAT systems, and to determine the values of key parameters. 

The following can be rendered inoperative: attitude g}TO, directional Oto, 
altimeter, airspeed indicator, turn coordinator, vertical speed indicator, 
VOR/LOC Indicators, automatic direction finder indicator, glldeslope, 
and engine. Additionally, oil pressure, oil temperature, cylinder head 
temperature, fuel level for each tank, engine sound volume, gross weight, 
center of gravity, outside air temperature, rough air magnitude, barometric 
pressure, and wind direction and velocity are subject to continuous control. 
An X-Y plotter connected to the GAT tracks the progress of the flight on an 
enroute low altitude chart, and provides the air traffic controller with the 
equivalent of radar flight monitoring. 

Communication channels have been wired to permit tw’o-way communica- 
tion between the Jumpseat^and the ATC monitoring station. It allows pilot 
activities such as frequency changes to be relayed to ATC by the Jumpseat 
observer as augmentation to video monitor viewing. It also provides for 
ATC cueing of Jumpseat for changes in environmental GAT parameter t^nd 
introduction of systems failures. 

^Jumpseat refers to an exporimenu r who rides outside the cockpit but who 
can both observe pilot cocKplt behavior and also Initiate system failures. 
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The equipment described above helps to provide fidelity and realism for 
the subject, adequate experimental control of the flight environment, and 
audio-visual recording of experimental flight data. 

C. FMS Procedures 

The following materials support a GAT scenario experimental session 
from initial contact of subject to raw data collection. T\plcally, three 
experimenters are required to execute a session. One (Director) handles 
subject contact before and after the simulated flight. Another acts as 
ATC during the run, and a third sits In the GAT Jumpseat to control 
cockpit conditions and to call out Instrument status. The general pro- 
cedure for a GAT scen:u*lo experimental session follows: 

1) Subject contacted, explanation of study read, appointment is 
made, aircraft manual and subject background data form are 
mailed to Subject. 

2) Subject arrives and is met by Director. Subject is taken to a 
briefing room, where he initiates fli^ii planning. 

3) Meanwhile, ATC prepares control station and Jumpseat pre- 
pares GAT with detailed checklists. 

4) Director prepares GAT room conditions and sees that all 
checklists are completed. 

5) Wlien Subject finishes planning, Director escorts him to GAT 
room and familiarizes Subject with GAT cockpit. 

G) Director has Subject start the engine and closes cockpit door. 
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Jumpseat controls engine status gauges, winds aloft, and 
various other environmental conditior-9 as cued by ATC or 
Director. Jumpscat also monitors and reads Inatrumeats that 
are difficult to read from the video camera. This aids In later 
review of the video tape and also aids ATC in determining vrhlch 
comm frequency has been selected. 

Director fills out data form for the particular run Including 
the clock times fer significant events to aid subsequent video 
tape reviews. He also obtains Subject performance judgments 
from ATC and Jumpseat at several points In the scenario. 

10) After Subject lands the aircraft, Director meets Subject In cock- 
pit and takes Subject to debriefing room. 

11) Subject discusses the flight with Director, answering specific 
questions concerning the CIFE. The debriefing Is recorded on 
audio tape. 

12) Meanwhile, ATC and Jumpseat shutdown GAT and supporting 
hardware, and document and store raw data. 

D. FMS Scenarios 

Three separate full mission scenarios have been created. Each scenario 
has accompanying support material in terms of charts, experimenter check- 
lists, ATC scripts, pre-recorded background communication tapes and data 
forms. Samples of these support materials are contained In Master Notebook. 
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Each of the three scenarios features a different tj-pe of crlticel In-flig^t 


event. Scenario 1 Involves a loss of fuel from one tank. Full pov/er Is 
j recoverable by switching tanl« at which time the pilot must decide on one of 

several destination alternatives. Scenario 2 Involves a partial power failure, 
j No actions are available to restore full power to the aircraft. The pilot must 

decide on one of several destination alternatives or an emergency landing. 

I I 

j Scenario 3 involves a partial navigation system failure during an ILS 

' I approach. The pilot must recognize the failure and select an appropriate 

' J alternate approach procedure and/or airport. All scenarios feature weather 

‘ near IFR mlnimums and a mix of mountainous ^ flat, and seacoast terrain. 

I Details of each scenario appear below. 

Scenario 1 

^ The objective of this scenario is to reveal how a pilot responds to inadver- 

tent loss of fuel in flight, resulting from the over-wing siphoning of fuel through 
' an improperly scaled filler opening. Of particular interest are, l)hls actions 

to restore engine power when the fuel supply from the tank in use Is depleted, 

2) his decision on where to land in view of the unanticipated reduction of 
remaining fuel and 3) his aircraft control performance prior to and after the 
CIFE. 

I Each subject Is instructed to prepare and file an IFR flight plan for a night 

■ flight from Seaport Beach to Mountaindale airport. The weather at the point 

of departure and along the route of flight is IMC (ceilings are less than 1000 
feet and visibilities arc less than three miles). At Uie destination airport 
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the weather is marg^inal V?.IC. Seaport Beach Is on the coast; Mountain- 
dale Is surrounded by mountainous terrain. 

After takeoff the flight proceeds along a predetermined route as specified 
In the original clearance (radar vectors to the Seaport 259° radial to Ranch 
Intersection, Victor 97 to Goathill VOH, direct). As the flight continues 
along this route, the pilot is Instructed to contact the appropriate controlling 
facilities. The fuel supply in the tank in use is reduced gradually, but at a 
rate much faster than that of normal consumption. When the flight reaches 
a certain point, the fuel supply in the tank being used Is depleted, and the 
engine sputters and dies. At the time of the engine failure, the flight Is In 
instrument conditions, experiencing moderate turbulence, and not In radar 
contact. 

The only action the pilot can take to regain engine power Is to switch 
fuel tanks. In the course of solving this problem, the pilot must set 
priorities concerning the activities he deems appropriate. Once the 
pilot switches tanks, and engine power is restored, normal operations 
ca»“ be continued. However, the flight now has half the original fuel. In 
view of this new limitation, the pilot must decide on whether to continue or 
to divert to an alternate. There are three alternatives from which the pilot 
must choose: he can continue on to his destination, land at a closer air- 
port, or return to the point of departure. The flight has fuel sufficient 
to fly to and land at any of the alternatives, but his choice is complicated 
by varying weather conditions at the different airports, the different distances 
and times to fly to die airports, and the pilot's perception of the problem. 
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Scenario 2 


This scenario simulates the reduction of available en^ne power due to 
a broken baffle in the muffler during a cross-country IFR flight. 

The mission is to fly from Seaport Beach to Mountaindale, Immediately 
prior to takeoff, the pilot is cleared along a route approximately parallel 
to the one which he had filed. At the time of departure the Seaport Beach 
weather is IMC (ceiling is liOO feet and visibility is tv/o miles in ruin, fog, 
drizzle) and tiie Ivlountalndale weather is marginal VMC. 

As the flight progresses, moderate turbulence is encountered near 
Singer intersection with a tailwind at thirty knots. When the flight pro- 
ceeds past Thermal intersection, engine power is linearly decreased to 
1500 rpm over a period of three minutes. This is accompanied by tachometer 
indications and a decrease in engine sound. Simultaneously cylinder head and 
oil temperature are increased to maximum level. The pov/er level is not 
sufficient to maintain the enroute altitude, so a descent begins as the power 
loss continues. The problem consists of inadequate power and rising terrain 
while out of radar contact in instrument conditions. 

At this point the available alternatives are: 1) continue to Mountalndale, 

2) return to Seaport Beach, 3) land on the immediate terrain, 4) land at 
Singer, 5) land at Wind FaP-:, 6) land at Link County, and 7) land at Pelton 
Naval Air Station. A major decision is whether or not to declare an emer- 
gency, especially since he assigned altitude cannot be maintained. Ti^lcally 
the subjects proceed to Mountalndale or return to Seaport Beach. 
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Scenario 3 

The purposes of this scenario are 1) to reveal how a pilot, without the 
aid of warning lights or flags, determines that an essential part of his 
approach navigation equipment (localizer) has failed, and 2) to reveal what 
decisions and actions he makes to complete the flight In view of the aircraft's 
new status. The pilot is Instructed to depart IvTountalndaJe airport, to con- 
duct two ILS approaches at Mountaindale, and to land at I^Tountalndale after 
the second approach. His flight plan specifies the route of flight to be *Via 
radar vectors**. The local weather conditions during depamre and the two 
subsequent instrument approaches are **celling 500 feet overcast, visibility 
two miles in rain and fog, wind from the east ai tea knots. *' 

After takeoff the pilot is vectored along a predetermined route to inter- 
cept the localizer course for the runway five ILS approximately five miles 
from the outer marker. After completion of the first approach, the pilot 
is vectored around to intercept the localizer for his second, and final 
approach. As tlie localizer needle sweeps to the center position during 
initial interception, it is rendered inoperative. (In this mode the localizer 
needle remains Idle in the center position with a *TO'* Indication. ) At the 
time of failure, the flight Is in instrument conditions, in radar contact, and 
experiencing light turbulence. 

The pilot can use ATC position Information, ADF crosschecks, or note 
that the needle is stationary to determine the localizer needle has failed. 

Upon confirming its failure, the pilot then must decide what to do next. He 
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could conduct an NDB approach, a VOH approach, or divert to another air- 
port. Of these alternatives, the VOH approach is the only feasible one. 


E. Subjects 

Twelve subjects were selected for the FMS experiments. Four were used 
in each of the three scenarios. Their ages ranged from 21 to 56 years old. 
Although all of the subjects were instrument rated, their licenses covered 
the spectrum from Private to ATP. Sb: of the twelve held CFI ratings and 
five held turbine ratings, in terms of their primary flying activities they were 
equally divided (six each) into pleasure and professional flying groups. Their 
total flying hours logged ranged from 270 to 8800 hours. Table V-1 summarizes 
these data. 


Table V-1 

FMS Subject Background 


Scenario 

2 hject 

Age 

Licenses 

Total Hours 

T\T?e of Flying 

1 

1 

36 

Pvt. 

420 

Pleasure 


2 

56 

Comm/ 

CFI 

5000 

Pleasure 


3 

42 

Comm 

1200 

Business 


4 

46 

ATP 

8800 

Bus iness 

2 

5 

23 

Comm/ 

CFI 

1550 

Professional 


6 

34 

Comm 

5000 

Pleasure 


7 

34 

Comm/ 

CFI 

3000 

Pleasure/Mil 


8 

30 

Pvt. 

300 

Pleasure 

3 

9 

31 

Comm/ 

CFI 

1750 

Professional 


10 

22 

Pvt. 

270 

Pleasure 


11 

21 

Comm/ 

CFI 

480 

Professional 


12 

21 

Comm/ 

CFI 

600 

Professional 
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As noted In Table V-i an attempt wa.s made to obtain a mix of exper- 
ience and ratings for each of the test scenarios. All subjects were unpaid 


volunteers from the Columbus, Ohio area. 

F. Data Collection 

As noted In Figure V 1, three major types of performance data v/ere 
collected for each FMS run. 

1) ^Stlck and rudder" performance, l.e. , basic control of heading, 
altitude, and airspeed 

2) Communications 

3) Response to the CIFE 

Stick and rudder performance was evaluated both objectively and sub- 
jectively. Subjective ratings on a scale of one to seven were given for navi- 
gation skills and attitude control by each of the three experimenters present 
during a run. (All experimenters were qualified pilots as well as researchers. ) 
Subjective rating averages for both navigation and attitude control skills ranged 
from a low of 1.2 to a hl^ of 6.7. There appeared to be a high correlation 
between the two ratings, l.e. , a subject with good navigation skl.'ls also 
exhibited good attitude control skills as noted In Figure V-2. Only ten subjects 
were rated due to unscheduled equipment malfunction during a portion of two 
runs. 


A more objective indication of stick and rudder performance was obtained 
from time traces of altitude, airspeed, and heading deviations covering the 
period Immediately surrounding the introduction of the CIFE. These data 
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Figure V-1 



were obtained by analyzing video tapes and coordinated audio tracks for 
each FMS run. Samples of each data are contained in Figures V-3 and V-4. 
Plots for all subjects are contained In Appendix F. 

CommunicatiOL skills were also evaluated both subjec;:ively and 
objectively. Each observer rated each subject on a scale of one to seven. 
Average scores here ranged from a low of 2.0 to a high ot 6.3. In addition 
to those ratings, which were made at the time of the experiment, complete 
transcripts of communications were prepared ^^ter the fact from the auilo 
tapes. A portion of one such transcript for the second scenario around the 
time of the CIFE has been reproduced in Figure V-5. These transcrip<^s per- 
mitted 3 detailed analysis ot interactions bel'ween pilot and controller as well 
as an indication of the Information search by subjects. 

The third indication of performance was the actual decision making 
response of subjects when faced with a CIFE. A standard data sheet was 
used to summarize the observed behavior of each subject. Problem defection 
and diagnosis as well as decisions and actions were noted (see Figure V-6). 
The information used to complete these sheets was obtained by studying 
the video tapes, consulting observers* data sheets and fx-om a thorough 
(tape recorded) debriefing of each subject after his FMS run. Observed 
stress was a subjective estimate scale one to ten) by die experimenters. 

Pilot criticality estimates were made by the subjects (scale one to ten) ana 
were intended to indicate the degree of criticality each placed on the CIFE. 
Flying time estimates were made by the subjects who were asked how long 
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Figure V-5. CommutiiCRtloas Transcript, Siibject 7, Scenario 2 
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15.59 

15.71 

16.23 


16.40 

16.71 

16.81 

16.96 

17.06 

17.17 

19.20 

19.33 

19.45 

19.52 

19.74 

19.84 

20.48 

20.60 

20.66 

20.73 

20.81 

21.96 

22.07 
22.80 

22.98 

23.05 

23.22 

23.31 

23.51 

23.59 

23.65 


23.85 

24.07 

24.31 

24.40 

24.82 


25.10 


25.41 


C: Japan Air 231, contact East Bay Center, 132.15 heading. 

B; Japan Air 231 to 132.15, good day. 

C: Uh, one niner two golf papa, roger, maintain 6000. 
(unintelligible). 

P: One niner two golf papa, roger, maintain 6000. 

P: Center, November one niner golf papa reporting vfhermal 
at this time. 

C: Two golf pop’s at TlierTnal intersection. Thank-you, sir. 

B; East Bay Center, United 694 climbing to one-two-zero. 

C: United 694, radar contact, clinb unrestricted to flight 
level three-seven-zero. 

B: Unrestricted to three-seven-zero. United 694. 

C: Uh, one niner two golf papa, radar contact, uh, 
(unirit.elligible) two miles north of Theimal intersection. 

P; Uh, roger, one niner two golf papa. 

C: Answer golf papa, you can expect, uh, 8000 in ten miles. 

P: Two golf papa, roger. 

C: King Aire SO Fox Hotel, contact Seaport Approach 119.6 
B: 119.6 fer SO Fox Hotel. 

B: East Bay Center, Centurion 5343 Foxtrot climbing to .7000. 
C: 43 Foxtrot. East Bay Center, ident. 

B; 43 Foxtrot, roger. 

C; 43 Foxtrot, radar contact, proceed on course. 

B: 43 Foxtrot, on course, roger. 

C: Pacer 62, contact Pel ton approach 126.2. 

B: Pacer 62 to 126.2, good day. 

B: Center, this is Baron 3622 Tango. Has anyone reported 
turbulence on vector two -twenty -two to the northwest here? 

C: 22 Tango, that's negative, sii".' 

D: We’re in moderate turbulence at 12,000, and picking up 
mixed ice. Any chance of one four thousand for 22 Tango? 

C: Baron 3622 Tango, climb and maintain one four thousand. 

B: 22 Tango leaving 12,000 £ot one four thousand. 

P: Center, November one nine two golf papa. 

C: One niner two golf papa, go ahead, sir. 

P: Okay, roger sir, experiencing ,uh, difficulties with 
my engine. I’m losing RPM and request, uh, inimediate descent 
to the nearest airport. 

C: One niner two golf pop, uh, stand by. 

C: Two golf pop, uh, all the airports in the vicinity are 
IFR. You're currently, uh, five miles northwest of Thermal. 
P: Uh, roger, what’s, uh, what^'s that weather at Link? 

C: Okay, uh, stand by. I’ll have it for you i.n just a second. 
C: Yeah, two .golf pop, uh. Link weather at zero .300. Uh, 

500 scattered measured GOO overcast two miles rain and fog. 
Uh, wind one six zero at ten, altimeter two niner point 
four five. 

P; Uh, roger, and Center be advised, uh, one niner two 
golf papa is, uh, losing altitude at this time, unable to 
maintain altitude, and RPM is dropping off. Uh, request 
vectors for the clearest weather possible you can find. 

I’ro gonna have to be setting it down. 

C: Uh, two golf papa, understand (unintelligible). Unable to 
maintain altitude, requesting vectors. All airports in the 
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Figure V-6. Scenai-lo 1 - CIFE Response 


j Detection mode 

hoard enyc. quit 

« 

heard enq:. quit 

heard eng. quit 

fuel (^uf^e 

I Detection time 

immediate 

immediate 

Immediate 

Immediate 

iDiagncstic Procedure 

v'l 

knew left 
tank was low. 
switched 
tanks 

fmmediatel}*' 

carb heat 
switched tanks 

carb heat 
throttle 
starter switch 
switched tanks 

switched 
tanks 
prior to 
engine 
failure 

1 Prepared st»'ategy 

none 

none 

none 

identify- alle 
before oeclar 

Declared emergency? 

1 

no 

no 

yes 

nc 

1 Info sources used 

none 

none 

none 

fuel gauge 

Perceived cause 

drained tank 

drained tank 

drained tank 

drained tank 

1 Observed stress 

1 

1 

8 

1 

Pilot's crit. est. 

5 

3 

9 

9 

Relevant experience 

same event 

same event 

lost 1 helo. eng:. 

sw. aux. to rt 

Est. flying time 

2 hrs. 

1. 6 hrs. 

1. 6 hrs. 

2 hrs. 


Alternatives 

considered 

Link Co, 
Seaport 

Link Co. 
Mount alndale 

Seaport 
Mountaindale 
Link Co. 

Mountaindale 
Link Co. 

Decision 

Return to 
Seaport 

Continue to 
Mountalndale 

Continue to 
Mountaindale 

Continue to 
Mountaindale 

Reasons 

Had flying time 
wanted to go 
home 

enough fuel 
ILS appr. 

not discussed 

not discussec 

Changes in plan 

none 

none 

Divert to 
Link Co. 

Divert to 
Link Go. 

Reasons 



VOH Approach 
good weather 

better distanc. 
given fuel & 
good weather 

Flying technique 

no change 

no change 

no change 

max. enduran 

Outcome 

Successful 
NDB with aid 
of ATC 

Successful ILS 
with aid of 
ATC 

Successful VOR 
with aid of 
ATC 

Missed VOR 
successful VC 
at Link Co. 
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Figure V-7* Decision Factor?! Rating 


Scenario #1 

1 

2 

3 

4 

Average 

Rang 

Estimated flying time 

•7 

6 

6 

1 

6.5 

2 

E St im riled fuel on board 

4 

4 

4 

4 

4.0 

1 

Enroule weather 

- 

5 

- 

3 

4.0 

3 

Destination weather 

- 

6 

1 

3 

3.3 

6 


Scenario #1 

1 

2 

3 

4 

Average 

Rang 

Conservatism 

4 

6 

6 

7 

5. 7 

4 

Safety 

6 

7 

7 

7 

6.7 

2 

Match with piloting skills 

7 

6 

7 

5 

6.2 

3 

Familiarity with A/C procedures 

1 

4 

7 

4 

4.0 

7 

Familiarity with NAV procedures 

1 

6 

6 

4 

4.2 

6 

Proximity to original intentions 

1 

7 

2 

3 

3. 2 

7 

Compliance with FARs 

1 

G 

6 

5 

4.5 

6 

Flying time needed to execute 

4 

4 

4 

3 

3.7 

2 


Scenario M ^ 


= 

1 3 

4 

Average 

Ran 

A/C condition when decision made 

1 

7 

3 

7 

4.5 

7 

Fuel on bo.'ird 

6 

6 

5 

7 

6. 0 

3 

Enroule weather 

1 

1 

2 

1 

1. 2 

2 

Destination weather 

5 

1 

7 

7 

5.0 

7 

Time needed to execute 

7 

7 

4 

7 

6.2 

4 

Familiarity with A/C procedures involved 

7 

4 

7 

5 

5.7 

4 

Familiarity with NAV procedures involved 

6 

7 

4 

5 

5.5 

4 

Proximity to original destination 

7 

7 

2 

7 

5.7 

6 

Compliance w'ith FARs 

1 

4 

1 

5 

2. 7 

5 

Other 

- 

- 

3 

- 

- 

- 
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they thought they could continue to fiy after the CIFE. The rest of the data 
sheet entries were filled by experimental observation or subject statement. 

In an attempt to probe their personal rationale* each subject was also 
asked to complete a rating form covering some 21 separate factors which 
may have influenced his decision In the face of tho CIFE. Each factor was 
rated by the subject on a scale of one to seven. (Ratings for Scenario 1 r.re 
shown In Figure V-7.) 

The final piece of InTormatloa collected was the test score from an open- 
form knowledge survey. This survey was used as a pilot study to help develop 
the closed-form knowledge survey used with the paper and pencil scenarios. 
All subjects for scenarios one and two also participated in the paper and 
pencil tests. They were identified In the master data sheet with ”1” in the 
GAT column. Complete data summary sheets for all three scenarios are 
contained in Appendix F. 


G. Performance Evaluation 

Because of the small sample size and differences across scenarios, 

It was difficult to develop solid statistical information concerning pilot per- 
formance in such full mission simulation studies. However, by analyzing 
the data mentioned above, It became apparent that tlie subjects in these 
experiments possessed a wide range of cockpit management styles and skill 
levels. Although difficult to quantify, ”good performance” was easily recog- 
nized by both on-site observers of the FMS runs and others who examined 
the various data collected from those runs. The elements of ”good performance” 
Included: 
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1) professional use of the radio 


2) precise heading and altitude control 

3) constant awareness of the aircraft position along its 
Intended route 

4) prompt, but not necessarily Instant, response to the onset of the 
CIFE (detection) 

5) systematic procedure for trouble-sbccting 

6) diversion decisions which allowed for further uncertainties 

Evidence supporting each of these six characteristics of good performance 

can be found In Figures V-2 to V-7 above. For example, consider Figures V-3 
and V-4 which depict what app>ear to be good and poor stick and rudder perform- 
ances. The time traces for subject 4 exhibit very small unplanned deviations 
in airspeed, altitude and heading both before and after the onset of the CIFE 
(loss of fuel cap). Subject 3, on the other hand, demonstrates a somewhat 
unstable control of these three flight parameters even before the onset of the 
CIFE. Furthermore, during and after the CIFE, his airspeed, altitude and 
heading excursions appear to increase in both frequency and amplitude which 
may indicate that he was loaded beyond his ability to cope with the problem at 
hand. Coincidentally, It is also easy to find evidence that subject 4’s per- 
formance in each of the six elements listed above was generally superior to 
that of subject 3. Furthermore, there is supporting evidence that the * good 
performers” tend to score higher on both forms of the knowledge survey than 
do the ^*poor performers”. 
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since much of the evidence of FIvfS performance is anecdotal, a brief 

! narrative description of each subject’s actions and their characteristics 

I has been prepared. These narratives are contained in Appendix F. These 

! 

narratives Include comments on each subject’s background, personal char- 

H acteristlcs, and management style. They are perhaps the richest information 

source for gaining Insights Into how these twelve subjects made use of 

I 

' 1 

available resources In the face of critical In-fllght events. 

• H. FMS Conclusions 

j The sample was too small to provide anything other than some Initial 

I hypotheses concerning pilot performance in such a full-mission setting. 

\ 

I 

' However, the following tendencies were noted; 

1) Cockpit management style varies widely among pilots. For 
example, some are extremely self-reliant, others want 
Immediate and extensive help from ATC while still others 
make the decision making process a joint effort with ATC. 

2) Good stick and rudder pilots seem to have excess capability 
and maintain good stick and rudder performance during and 
after the CIFE. More marginal stick and rudder pilots, on 
the other hand, show Increased frequency and amplitude of 
heading and altitude excursions, and experience communication 
difficulties in the face of a CIFE. 
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3) Pilots who score well on the knowledge test instruments 
tend to perform well in problem diagnosis and decision 
making. (GAT subject performance on the paper and pencil 
tests are discussed in Section VT-L.) 

From the observations of the experimenters and comments made by 
participating subjects. It appears that such a full mission simulation exer- 
cise, coupled with an appropriate Imowledge survey ar.d debriefing, could 
be a valuable tool for recurrent training of IFR pilots. 
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VI. PAPER AND PENCIL SCENARIO TESTS 


I 

The GAT FMS scenarios were extremely valuable in gaining a better 
I understanding of how pilots make decisions in the face of CIFE*s. However, 

they were very expensive to run, in terms of equipment, subject and exper- 
imenter time, and the data was difficult to analyze in objective fashion. The 
paper and pencil scenario concept was developed to provide a more econom-- 
I ical way to study the CIFE phenomenon and to reduce the data collection and 

* analj'-sls problems inherent with FMS experiments. 

f 

A. Background 

! The paper and pencil (P/P) concept was tested in two different ways prior 

to full-scale Implementation. First, two pilots, both on the aviation faculty 

at The Ohio State University and considered to be experts in their field, 

evaluated several GAT subjects* decisions on two of llie three GAT scenarios. 

The two experts then made their own diagnoses and decisions on the third. 

From these sessions it became clear that pilots could diagnose problems and 

make diversion decisions in a p/p format. Furiherm.ore, the expert pilots 

I found the tasks more realistic when they injected themselves into the 

scenario, rather than playing the role of observer. 

* A second P/P format pre-test was run with a local aircraft mechanic who 

is widely respected as an expert. The purpose of this exercise was to determine 

I 

if someone could diagnose a mechanical failure in an interview situation. The 
mechanic was given the Initial symptoms to the problem and was asked to arrive 
at an explanation of the cause. He asked questions about the status of various 
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Indicators and hypothesized aloud as he systemailcally eliminated potential 
causes. The Interviewer provided readings from instiniments and answers 
to sundry status Inquiries verbally. The mechanic had no trouble diagnosing 
the problem in the Interview format. These results suggested many of the 
techniques used in the full-scale study. A transcript of part of that interview 
is contained in Appendix K. 

In order to facilitate analysis and to eliminate interactions, it was decided 
to break the paper and pencil testing into two distinct elements; one set of 
scenarios directed toward problem diagnosis and a second set directed toward 
pilot decision making based upon a common diagnosis of the problem. 

The diagnosis scenarios were conceived to meet several important criteria: 

a) a system or component failure that would be nondeteriorating over time, 

b) insoluble (at least while in the air), but identifiable, c) precipitated by com- 
ponent failure or weather conditions, and d) Important enough to require a 
subsequent diversion decision. There also had to be enough evidence within the 
available Information to unambiguously identify the cause of the problem. 

Once the four problems were selected for use, the concomitant sj^mptoms 
and instrument readings were verified with the expert airplane mechanic 
referred to earlier. The given symptoms for the problems were selected to 
lead the subject in the general correct diagnostic direction, but were Insufficient 
for trivial solution. The four scenarios selected Involved: 1) an oil leak at 
the oil pressure gauge line, 2) a vacuum pump failure, 3) a magneto drive 
gear failure, and 4) a frozen static port. The diversion scenarios designed to 
illuminate a pilots decision making strategies are discussed In Section E below'. 
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B. The Testino: Procedure 

The procedure used in the paper and pencil scenario (PPS) testing required 
about ninety minutes. The period was used for four major data collection inputs: 

a) Biographical Data (See Appendix K) 

b) Closed-Form Knowledge Test (See Chapter IV) 

c) Diagnostic Performance on Four Different Scenarios 

d) A Destination-Decisicn Problem Dealing With Information Seeking 
Strategies 

These will be discussed in detail in the following sections. 

Announcements were posted at local flying clubs and fixed base operators 
(FBOs) to attract volunteer subjects from the flying community. Interested 
IFR rated pilots called in for details and were scheduled for one of several 
two-hour sessions. In addition, qualified pilots from The Ohio State University 
and local communities were called by telephone and jn/ited to participate. 

Each session proceeded as follows: Participants gathered in a large con- 
ference room. After a brief introduction by one of the principal Investigators, 
subjects filled out the biographical forms and took a clcsed-form knowledge 
survey. A briefing statement covering scenario weather, airspace and the 
airplane to be ^’flown” was given the subjects while they looked at enroute 
charts and weather maps (see Appendix H). The subjects then went individually 
to separate rooms with an experimenter. Here, they went through the problem 
diagnosis and diversion-decision excercises for about one hour. 

The instructions were read to the subjects (see Appendix H) which e^q^lained 
how the four problem diagnosis scenarios would be run. For each problem 
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diagnosis scenario, a brief mission introduction was read, identifying area 


i I 


weather* flight origin and destination, and referring to a low altitude enroute 
chart with the airports highlighted. Following the introduction, symptoms 
for the problem were given (e.g. ’’After twenty minutes of routine flying you 
notice the smell of hot engine oil”). At this point the subject was signalled 
to begin his diagnosis by the question, ”Wliat would you do?”. A stop-watch 
was started when the subject began his information search, allowing four 
minutes for completion. 

While referring to a modified diagram of the Piper Arrow instrument 
panel, subjects began to ask the experimenter for pieces of information which 
could be collected by the pilot if he were actually in the cockpit of a Piper 
Arrow. In addition to readings from flight Instruments, engine gauges and 
navigation/communication radios, the subject could query the experimenter 
for information concerning structural ice formacion, noise, cabin conditions, 
status of the cabin interior, and system response to control settings 
for throttle, mixture, RPM, fuel selector, etc. WTien queried, the experi- 
menter looked up the information on two sheets of paper which followed a 
standard format. After finding the lequested information and telling the 
subject, the experimenter then noted the Item with a number on the sheet. 

The numbers denoted the sequence of queries such that the order could be 
reconstructed. A third sheet was available for noting hypotheses of potential 
causes mentioned by the subject during the information search. Their position 
in the sequence was also noted. 
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The clock vyas stopped when the subject indicated that he had discovered the 
cause of the problem- If the four minutes ran CAitf the subject was asked to 
make a best guess as to the problem’s source. The time taken was 
and the subject was asked how long he thought the plane would fly with inu h a 
problem, lie was then asked to Judge 'he criticality of the problem as he I ad 
diagnosed It* on a scale oi one to seven. An explanation of the cause of the 
problem was then read to the subject and rhe final two estimates were repealed. 
This procedure was repeated for four different scenarios and took about 
25 minutes to complete. 

Forty volunteer subjects participated in the P/P scenario study. All 
but one were instrument-rated and with experience ranging from 160 to 
19,400 total flying hours. Nineteen had commercial licenses end twelve had 
Air Trarjsport ratings. Eight of the subjects had participated earlier In the 
GAT-1 study. Subject background data is shown in Table VI- 1. Figures 
\T-1 to \T-3 depict the flight experience of the subjects. Table vH[-2 summarizes 
subject data for background data by frequency ana percem. Figure 11-4 depicts 
the subjects scores on the ciosed-form of the knowledge tesi. It is worth 
notlncr that the scores were surprisingly low considering the fact that the 
mean number of hours experience was 3823 hours. 


C. Pilot Background Data and Dlaenostic Data Collection 

Pilot background data were coded Into seven variables. The four continuous 
numeric variables were; score on the knowledge survey (0-20), total flying 
hours, total sfngle-englne hours, and total (Including acrual, simu- 
lated ..nd time flown under IFR) hours. Tlirce discrete variables were: 
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Table VI-2 


Frequency Analysis of Pilot Background 

Frequency 

Rating; 


Private 9 

Commercial 19 

Air Transport 12 

Training; 

Military 10 

Civilian 30 

Most Frequent Flying: 

Airline 2 

GA Commercial 12 

Business 11 

Military 6 

Pleasure 9 


Percent 


22.5 

47.5 
30.0 


25.0 

75.0 


5.0 

30.0 

27.5 
15.0 

22.5 


rating (private, commercial, and ATP), primary flight training (military or 


civilian) and the most frequent type of flying (airline, GA comm, business, 
military, and pleasure). 

Diagnosis scenario performance was coded Into ei^t numeric variables 
for each subject on each scenario. These were: 


I 

number of Inquiries or ocntrol actions 

TT 

total tracks (lines of coherent questioning) 

UT 

unique tracks (tracks not repeated) 

C 

correctness of final diagnosis (C-5) 

Z 

correctness/total tracks 

E 

efficiency = 25-2 x (minutes required) - 1-2 

CORINQ - 

correctness/total Inquiries 

M 

merlt*^C x E) 

CB 

criticality estimate before solution given 

CA 

crltlcalit>' estimate after solution given 

CNTRL - 

number of control actions taken 


The totals for these eight variables, summed across the four scenarios were 
also calculated and named as variables: 


TOTIKQ 

- 11 + 12 + 13 14 

TOTTR-‘\KS - 

• TTl TT2 ^ TT3 TT4 

TOTUTRKS - 

. UTI ^ UT2 UT3 UT4 

TOTCOR 

. Cl C2 ^ C3 - C4 

ZT 

. TOTCOR TOTTRAKS 

TOTEFF 

• El ^ E2 - E3 - E4 
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CORINQT 

TOTMERIT 

TCRITBEF 

TCRITAFT 


totcor/totinq 

Ml + M2 + M3 + UA 
CBl + CB2 + CBS + CB4 
CAl + CA2 + CA3 + CA4 


See the Glossary, Table Vl-3. Tlie diagnostic data, knowledge scores, 
pilot background data and decision data (see Section E below) were compiled 
auci used in the total analysis. 


D. Diagnostic Performance 

Means and standard deviations for all performance variables are listed 
in Table VI-4. Comprehensive scores of total correct and total merit are 
shown as percentage distributions in Figures VI-5 and VI-6. The total correct 
distribution appears somewhat negatively skewed, while that for total merit 
appears to be normal. 

Group performance on the four scenarios improved in terms of correctness 
and merit with the order of presenta.tion, although all four problems were 
judged to be equally difficult to diagnose. This fact demonstrates some 
learning and strategy development by the subjects. 

When the pilot sample is broken down by rating, several differences emerge 
on various diagnosis performance dimensions (see Table VI- 5). Total correct 
and total merit scores Increase as the level of certification goes up (Pvt. , 

Comm. , ATP) consistent with conventional wisdom. Performance on scenario 1, 
(the oil leak) seems to run counter to presumed knowledge by the higher certifi- 
cate holders. However, scenario 2 (vacuum pump failure) and 3 (magneto 
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Table \nf-3 j| 

Glossar y 

.1 

!• AGE: Age of the subject - categorized Into Intervals: 

1) age 30 yrs. - r 

2) 30 yrs. < age < 50 yrs. \ 

3) age > 50 yrs. 

■ T 

2. AIRPORTS: Airports the pilot was willing to pass to locate proper repair ^ [ 

facilities. 

3. AP: Variable for airports used in computer runs valued (0) if airports < 2 ; 

and (1) if airports > 2 . 

4. APP: Approach attribute of an airport. Includes ILS vs. NDB approach. : 

5. ATC: Air Traffic Control attribute of an airport (presence of radar). « 

Importance assessment of approach attribute of an airport. 

7. Pilots importance assessment of an air traffic control attribute of 
an airport. 

8. Pilots importance assessment of time. 

9. B^y^: Pilots importance assessment of weather. 

10. Cl: Correctness score on Scenario #1 (possible correct: 0-5). 

11. C2: Correctness score on Scenario #2 (possible correct: 0-5). 

12. C3: Correctness score on Scenario #3 (possible correct: 0-5). 

13. C4: Correctness score on Scenario i»4 (possible correct: 0-5). 

14. CAl: Subjective criticality estimate of event in Scenario #1 after being pro- 

vided with the answer (scale 1-7; l=lowest criticality). 

15. CA2: Subjective criticality estimate of event in Scenario #2 after being pro- 

vided with the answer (scale 1-7; l=lowest criticality). 

1C . CAS: Subjective criticality estimate of event in Scenario **3 after being pro- 
vided with the answer (scale 1-7; l=lowest criticality). 
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17. CA4: Subjective criticality estimate of event in Scenario #4 after being 


r 

18. 

provided with the answer (scale 1-7; l=lowest criticality). 

CATSCRl: First category score on knowledge survey - knowledge sub- 
score for engine and fuel systems (possible correct: 0-7). 


19. CATSCR2: Second category score on knowledge survey - knowledge sub- 

score for electrical systems and cockpit instrumentation 
(possible correct: 0-7). 

20. CATSCR3: Third category score on knowledge survey - knowledge sub- 


1 

score for weather and IFR operations (possible correct: 0-6). 

21. 

CBl: Subjective criticality estimate of event in Scenario #1 before being 
provided with the answer (scale 1-7; l=lowest criticality). 

[ 22 . 
1 

CB2: Subjective criticality estimate of event in Scenario #2 before being 
provided with the answer (scale 1-7; l=lowest criticality). 

23. 

I 

CB3: Subjective criticality estimate of event in Scenario #3 before being 
provided with the answer (scale 1-7; l=lowest criticality). 

24. 

CB4: Subjective criticality estimate of event in Scenario #4 before being 
provided with the answer (scale 1-7; l=lowest crlticalit}')* 

25. 

CNTRLl: Number of inquiries which involved control movements in 
Scenario #1. 

26. 

CNTRL2: Number of inquiries which Involved control movements in 
Scenario #2. 

27. 

CNTRL3: Number of Inquiries which Involved control movements in 
Scenario #3. 

28. 

CNTRL4: Number of inquiries which Involved control movements in 
Scenario #4. 

29. 

CNTRLTOT: Total number of inquiries for all four scenarios which 
involved control movements 

CNTRLTOT = CNTRLl + CNTRL2 + CNTRL3 + CNTRL4 

30. 

CORINQl: Ratio of correctness to Inquiries for Scenario #1: 
CORINQl = Cl/ll 

31. 

i 

CORINQ2: Ratio of correctness to inquiries for Scenario #2: 
CORINQ2 = C2/I2 

1 • 
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32. C0RINQ3: Ratio of correctness to Inquiries for Scenario #3: 

CORINQ3 = C3/I3 

33. CORINQ4: Ratio of correctness to inquiries for Scenario #4; 

COPJNQ4 = C4/I4 

34. CORINQT: Ratio of total correct to total inquiries for all four scenarios: 

CORINQT = (Cl + C2 + C3 + C4)/ai ^ 12 + 13 + 14) 

Change In subjective criticality estimate of event for Scenario #1 
after being provided with the answer; DELTi^Cl = CAl - C31 

Change in subjective criticality estimate of event for Scenario #2 
after being provided with the answer; DELTAC2 = CA2 - CB2 

Change in subjective criticality’ estimate of event for Scenario #3 
after being provided with the answer; DELTA C3 = CA3 - CB3 

Change in subjective criticality estimate of event for Scenario #4 
after being provided with the answer; DELTAC4 = CA4 - CB4 

39. DIFl: Difference between number of total tracks and number of unique 

tracks in Scenario #1: DIFl = TTl - UTl 

40. DIF2; Difference between number of total tracks and number of unique 

tracks in Scenario #2: DIF2 = TT2 - UT2 

41. DIF3: Difference between number of total tracks and number of unique 

tracks in Scenario #3: DIFS = TT3 - UT3 


35. DELTACl: 

36. DELTAC2: 

37. DELTAC3: 

38. DELTAC4: 


42. DIF4: Difference betw^een number c ' total tracks and number of unique 

tracks in Scenario #4; DIF4 • TT4 - UT4 ^ 

43. DIPT: Difference between number of total tracks and number of unique 

tracks in all four scenarios: DIFT = TOTTRAKS - TOTUTRKS 

44. El: Efficiency score on Scenario #1: El = [25 - 2 (minutes to diagnose) - (II -2)] 

' 45. E2: Efficiency score on Scenario #2: E2 = [25 - 2 (minutes to diagnose) - a2 -2)] 

46. E3: Efficiency score on Scenario w3: E3 = [25 - 2 (minutes to diagnose) - (13 -2)1 , J 

I t 

1 ; 

47. E4: Efficiency score on Scenario #4: E4 = [25 - 2 (minutes to diagnose) - (14 -2)] 

I 

I ^ ’ 

48. FLY: Computer variable for the variable flying; takes values: i ' 

(0) if flying = 1,2,3, or 4 = non-pleasure 
I (1) if flying = 5 = pleasure 
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50. 


51. 


52. 


53. 


54. 


55. 


56. 


57. 
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60. 


61. 


62. 


63. 


FLYING: Most frequent kind of flying. 
Valued: (1) Airline 

(2) Commercial 

(3) Business 

(4) Military 

(5) Pleasure 


GAT: Participation in general aviation simulation; 0 = did not participate, 

1 = did participate 

GATKl: Open ended knowledge test on GAT subjects - subscore on engine 
operations (possible correct: 0-7). 

GATK2: Open ended knowledge test on GAT subjects - subscore on fuel 
systems (possible correct: 0-7). 

GATK3: Open ended knowledge test on GAT subjects - subscore on electrical 
systems (possible correct: 0-7). 

GATK4: Open ended knowledge test on GAT subje<.!:s - subscore on coclqjit 
Instrumentation (possible correct: 0-7). 

GATK5: Open ended knowledge test on GAT subjects - subscore on weather 
(possible correct: 0-7). 

GATK6: Open ended knowledge test on GAT subjects - subscore on IFR 
procedure (possible correct: 0-7). 

GATKT: Average of all parts of open ended knowledge GAT test: 

GATKT = GATKl GATK2 GATK3 + GATK4 -»• GATI^5 + GATK6 

6 


GONOGO: Designates whether the pilot would have taken the flight under the 
given conditions. Valued: (0) - would not go, (1) - would go. 

II: Number of inquiries in Scenario #1. 

12: Number of inquiries in Scenario #2. 

13: Number of inquiries In Scenario #3. 

14: Number of inquiries in Scenario #4. 

IFR: Variable designating upper and lower quartiles of IFR hours: 

(0) if IFR hrs. < 175 

(1) if IFR hrs. ^700. 
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64. IFRHRS: Hours of flying under Instrum^^^nt flight rules. 

65. INPTRl: Ratio of Inquiries to total tracks in Scenario #1: INPTRl = Il/TTl. 

66. INPTR2; Ratio of inquiries to total tracks in Scenario #2: INPTR2 = I2/TT2. 

67. INPTR3: Ratio of inquiries to total tracks in Scenario #3: INPTR3 =•* I3/TT3. 

68. INPTR4: Ratio of inquiries to total tracks in Scenario #4: INPTR4 = I4/TT4. 

69. INPTRT: Ratio of total inquiries to total tracks for all four scenarios: 

INPTRT = TOTINQ/TOTTRAKS 

70. KNOW: Variable designating upper and lower quartiles of KNOWLEDG scores: 

(0) if KNOWLEDG 9 

(1) if KNOWLEDG 5: 16 

71. KNOWLEDG: Score on aircraft systems survey (possible correct: 0-20). 

72. LATELY: Relative amount of flying done in last year: 

(0) if pilot has more than 50 hours 

(1) if pilot has less than 20 hours 

73. Ml: Merit score on Scenario #1: Ml = (Cl) x (El). 

74. M2: Merit score on Scenario #2: M2 = (C2) x (E2). 

75. M3: Merit score on Scenario #3: M3 = (C3) x (E3). 

76. M4: Merit score on Scenario #4: M4 = (C4) x (E4). 

77. MECH: Mechanic: (0) = not a mechanic, (1) = mechanic. 

78. PROPCONl: Proportion of control movements to inquiries in Scenario #1: 

PROPCONl = CNTRLl/ll 

79. PEOPCON2: Proportion of control movements to inquiries in Scenario #2: 

PROPCON2 = CNTRL2/I2 

80. PROPCON3: Proportion of control movements to inquiries in Scenario #3: 

PROPCON3 = CNTRL3/I3 


81. PROPCON4: Proportion of control movements to Inquiries in Scenario #4: 

PROPCON4 = CNTRL4/I4 

82. PROPCONT; Proportion of total control movements to total inquiries ii. all 

four scenarios; PROPCONT = CNTRLTOT/TOTTNQ 
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83. RAT: Cub6tl^ate variable for RATING used to plot Initial data tables. Takes 

on same values ns RATING. 

84. R.4TING: Rating type - 

1 - Pj-lvate 

2 “ Commercial 

3 “ Air Transport 

85. RATSCORE: Variable dividing ratings Into two groups - 

0 If private pilots (RATING " 1) 

1 If commercial or air transport pilot (RATING “ 2 or 3) 

86. RECENCY: Relative amount of flying time In past year - 

1 ” more than 50 hours 

2 “ betiveen 20 and 50 hours 

3 " Ices than 20 hours 

87. S: Specific subjects Involved la the GAT experiment - 

0 for subject numbers 11, 31, 32, 33 

1 for subject numbers 28, 34, 35, 38 

88. SEHRS: Hours of flying In a single engine aircraft. 

89. SEHRSLCXj: Natural logorlthm of single eriglne flj'lng hours: 

SEHKSL(X3 = LOGe (SEHRS) 

90. SHRSRANK: Variable designating upper and lower quartllcs for single 

engine hours: 

0 If SEHRS s 488.75 

1 If SEHRS £2075.25 

91. SUB: Variable dividing subjects - 

0 if subject number Is £ 30 

1 If subject number is > 30 

92. SUBJECT: Subject number (N ® 40) 

93. T: V'arlable designating upper and lower divisions ;or the variable TIM; 

0 If TDt < .625 

1 if TIM > 1 

94. TC: Variable designating upper and lower quartlles ot TOTCOR: 

0 If TOTCOR < 10 

1 if T(DTCOR £ 17 

95. TDELTAC: Sum of the changes in subjective criticality estimates for all 

four scenarios; TDELTAC = TCRITAFT - TCRITBEF 
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96. TE; Vorlable tie upper and lower quartlles of TOTEFK; 

0 If TOTEFF s 42 

1 If TOTEFF C59 

97. THnSLOG: Natural logarithm of totaly flying hours; 

TllRSLOG - LOGjp (TOTH IIS) 

98. THRSKANK; V’arlable desiggiatlng upper and lower quarttles for total 

flying hours: 

0 If TOTIIRS -s 1007 

1 If TOniRS 2 53*75 

09. TIM; Time attribute of an alternate airport - flying time to the airport 

100. T^’ : V'arlable designating upper and lower quartlles for total merit: 

0 if total merit s 129.25 

1 If total merit 2 235 

101. TOTCOR: Total correct score for all four scenarios: TOTCOH “ Cl * C2 + C3 ♦ C4 

(possible correct * 0-20). 

102. TOTCRIT.AFT: Total of subjective crltlcallt>' estimates for all four scenarios 

after being provided with the answers; 

TCRmFT = CAl + CA2 + CA3 * CA4 

103. TCRITBEF: Total of subjective criticality estimates for all four scenarios 

before being provided with the answers: 

TCRITBEF “ CBl -• CB2 * CB3 + CB4 

104. TOTEFF: Total efficiency score for all four scenarios: 

TOTEFF * El - E2 + E3 * E4 

105. TOTHRS: Total flying hours. 

lOG. TOTINQ: Total number of Inquiries for all four scenarios; 

TOTIN'Q = II * 12 * 13 + 14 

107. TOT^'ERIT; Total merit score for all four scenarios; 

TOTMERIT =* Ml + M2 M3 - M4 

103. TOTTRAKS: Total number of tracks for all four scenarios: 

TOTTRAKS = TTl * TT2 + TT3 - TT4 

109. T(DTUTRKS: Total number of unique tracks for all four scenarios: 

TOTUTRKS “ UTl UT2 * UT3 * UT4 
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110. TRA: V’arlable used to plot the TRAINING values la the data tables: 

1 = military 

2 ■ civilian 

111. TRAINING: Type of training (military or civilian). 

112. TTl: Total number of tracks in Scenario *1. 

112. TT2: Total number of tracks In Scenario #2. 

114. TT3: Total number of tracks in Scenario #3. 

li.'l. TT4: Total number of tr:icks In Scenario «4. 

116. I’Tl: Number of unique tr:icks In Scenario «*1. 

117. UT2: Number of unique tracks in Scenario «2. 

113. UTS: Number of unique tracks in Scenario *3. 

119. UT4: Number of unique tracks in Scenario «4. 

120. WX: Weather attribute of an alternate airport: Includes ceilings and vlsIbilUics. 

121. YOUNGOLD: Variable designating the upper and lower divisions of the age 

category: 

0 if age <- 30 

1 if age > 50 

122. Zl: Ratio of correctness to total tracks for Scenario #1; 

Z1 = CL'TTl. 

123. Z2: Ratio of correctness to total tracks for Scenario «2; 

Z2 “ C2/TT2. 

124. Z3: Ratio of correctness to total tracks for Scenario #3: 

Z3 = C3/TT3. 

125. Z4: Ratio of correctness to total tracks for Scenario *4: 

Z4 - C4 'TT4. 

126. ZT: Ratio of total correct fo total number of tracks for all four scenarios; 

ZT = (Cl ♦ C2 • C3 ♦ C4 )mTT1 ♦ TT2 * TT3 + TT4) 


i:;r 


o 

oo 


«>o 


c o 

■ i'o o ' 

o 






o 


o 

C5 '• 


69 



c. C 


o 


o 


o 


^ o 
® o 




o, 


o 


O ' 



drive gear failure) do demonstrate the monotonle relationship to rating* one 
would expect. Scenario (frozen static port) shows mixed results. 

Five aubjecls hold aircraft mechanic licenses (A and P) as well ns pilot 
licenses. In terms of gross measures, the mechanics* performances are 
superior to the other They have the top scores for knowledge, total 

correct, total merit and total efficiency. The only Inconsistency again shows 
up In sicennrlo A (frozen static ix>rt). However, since that problem relates 
to symptoms more likely to bo directly observed In their role as pilot rather 
than mechanic that result Is not totally unexpi'Cted. More extensive analysis 
of the diagnostic performance data will follow in section K. 

F. Decision Making Phase of P 'P Scenarios 
Procedure 

The decision making phase of the paper and pencil exercise was divided 
Into two parts: an information seeking part and a rank ordering of alternatlcos. 
The goal of the study was to determine the t>T>e of decision rule a pilot would 
use In a given problem, and to determine his worth structure concerning tlie 
characteristics of alnx>rts to which he might divert if it became necessary. 

The decision making portion of the exportmenl was bogian after the pilot had 
completed all four scenarios In the diagnosis v^hase described above. 

One basic scenario was used throughout the decision making phase and Is 
given In Apivndix H. The mission the hypothetical flight was to fly from 
Bangor, Maine, to Glens Falls, Ntw York, for a business meeting. The 
flight was to be made In a Chorokt^o Arrow and the weather at the time of the 
night, both along the route and at the destination, was marginal. Though there 
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Table VI- 4 (con’t.) 
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4.6OvCOCiO0 

0 

c 

3.©ooc:^80^ 

2.630035C9 

0 

IO.OOC80OOD 

2.0OOCCCC0 

0 

s.eococooo 

i.cococe &0 

0 

3.00003^0/00 

i.ococacoo 

d . 

2.6C303CC0 

1.25003000 

0 

C.GCCOOCGO 

1.0C033S00 

0 . 12500200 

2.60OOG00O 

0.38095236 

0 

1.66666667 

0.43454544 

0 

1.66666667 

0.44949490 

0 

1.66666667 

0.05555334 

0 

1.66666667 

8.38383683 

e. 04166667 

1.00000000 


n 

40 

40 

40 

40 

40 

40 

40 

40 

40 

40 

30 

40 

40 

40 

30 

40 

40 

40 

40 

40 

30 

40 

40 

40 

CO 


Table VI -4 (con*t.) 


BACia;rJ)U?S0. DIACKOSIS, and TOTAf-S 


H£AH 

biU DEV 

miuiAH 


KAXinUfI 

40.07500000 

30.211191260 

32 . CC 00 0000 

0 

IGB.GCGOGGCO 

33.63000000 

37.2I2C6654 

45.00050000 

8 

! le.ocoocaoo 

46.96000C00 

40.72156U66 

42.30000509 

0 

1 I3.6G53C380 

50.32300000 

33 .02636 IC4 

50.3eOOOCC3 

0 

1 1 0 • €OG’5'5O08 

176.45000000 

77,64331 190 

132.00500008 

10.05500098 

339.0(5003008 

5.05000000 

1. 75330500 

5.50005000 

1.60050500 

7.G<)CCK108d 

3.67300000 

2. 17663351 

3.50003000 

1.0O35COO9 

7. 0005^338 

4.350000(30 

a.6416G647 

4.50000000 

i.eooccoeo 

7.09035^38 

2.70300000 

1.6a24E900 

2.00003000 

i.COCO306O 

7.005505^8 

13.77300000 

3.99029271 

is.socooaoo 

B. 60300508 

24.00000558 

3.23076923 

1.356163112 

3.05300530 

1.05903038 

7.08033038 

3.52300000 

1.G2556630 

3.00000000 

1.O0COCCO8 

7.OCO0C058 

3.72500000 

1.60103137 

4.C0500300 

1.08000308 

7.00030308 

2.37500000 

1.40333625 

2.OCO5303O 

i.GOOOo^eo 

7.C0005C88 

12.07179437 

4. 12425105 

13.00005000 

4.0O5&55O8 

22.CC3O0OO8 

0.73OOOOOO 

:. 19292733 

0 

0 

4.00000008 

1.07500000 

1.13077349 

2.00030000 

9 

5.00300C4MI 

3.47500000 

1.67924739 

3.000^855(9 

1.CCOOO038 

B.03OO38CO 

1.93000000 

0.93232546 

2.00003008 

0 

4.03003088 

0.03000000 

2.046049119 

li.G00C9903 

3.03000506 

14.C0000000 

-1.76923077 

2.O7OS1570 

-I.O03C338O 

-6.60000008 

2.00000080 

-0. 15000000 

1.29190714 

0 

-4.O03&SOe» 

3.G000CO80 

-0.02300000 

1.734112431 

0 

-5.00038090 

4.0OOOGC88 

-0.32500000 

1.22762017 

6 

-5.00300009 

2.08000800 

-2.794U7I79 

2.7U321555 

-3.00003000 

-10.00050096 

2.900D0C00 
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Figure VI-5 
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Table \a-5 




f 

1 Diagnosis Performance Means by Rating 



Total Population 

PVT 

COMM 

ATP 

A & P 

Score on knowledge survey 

12.5 

10.5 

12.3 

14.1 

14.6 

Correct on Scenario 1 

2.7 

3.2 

2.8 

2.0 

3.4 

Correct on Scenario 2 

2.9 

2.7 

2.S 

3.0 

3.0 

Correct on Scenario 3 

3.4 

2.2 

3.4 

4.1 

4.0 

Correct on Scenario 4 

4.0 

4.1 

3.8 

4.3 

4.0 

Total correct 

13.0 

12.3 

12.9 

13.4 

14.4 

Total merit 

176.0 

156.0 

179.0 

186.0 

203.0 

Total efficiency 

50.0 

45.0 

51.0 

50.0 

53.0 


I 
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was no severe weather forecast (In the form of thunderstorms, turbulence, 


or ice), the prevailing rainy and drizzly conditions req»iired the flight to be 
conducted under Instrument Flight Rules (IFR). In fact, weather conditions 
were such that the flight would be in instrument meteorological conditions 
for almost the duration of the time aloft. 

After the pilot was given a brief Introduction on the mission of the flight, 
the navigation chart of the area, and the airplane, attention was turned to 
analyzing the weather in detail and filing a flight plan. The pertinent 
weather information was given to the pilot in a text written In ordinary 
English. This text is given in AppsudLx H. The wording of the text was 
intended to reproduce what one would normally hear in a telephone conversa- 
tion with a weather briefer. All of the weather information needed to plan 
the flight v as included. After the pilot confirmed that he had read and under- 
stood all of the weather, the next step was to compute and file a flight plan. 

In order to save time, the flight plan in Appendix H had already been com- 
piled based on the reported weathei’, and was shown to the pilot for his approval. 
The most important features of the flight plan were reviewed by the pilot, 
including flight routing, cruising altitude, and estimated time enroute. 

After reviewing all of the information on the airplane, weather, and flight 
plan, the pilot w'as asked if he would normal!ty attempt a flight under the stated 
conditions. He was also asked if there w'as any other information be wr>uld 
like to have concerning the proposed flight. His responses to these questions 
were recorded, and it w^as then time to embark on the flight. 
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The descriptlcn ot the flight started with a routine lift-off from die Bangor 

[ 

i Airport at the planned departure time. Climb-oat after departure and the 

I transition to cruising flight was uneventful. The pilot followed the progress 

of his flight on a very simplified enroute navigation chart (see Appendix H), 

I 

which portrayed the Intended route of flight, radio navigation aids and fbces, 

I and the departure and destination airports. 

The flight continued uneventfully until a point about midway along the route. 

\ At that point the aircraft encountered a serious problem with its electrical 

/ system. The problem was investigated (in the text) and was determined 

i 

to be an incperailve alternator. After the problem was clearly defined, an 
I 

^ upper time limit estimate was provided to indicate hov/ long the aircraft’s 

systems could rely on the reserve electrical power of the battery. 

I 

f 

The section of the ,ext in which the problem wap introduced and discussed 

j contained several key pieces ot information for the pilot. First, the symptoms 

t and the diagnosis set the stage for the need to divert. The straightforward 

\ 

statement of the diagnosis was intended to give each pilot, basically, the | 

I 

! same perception of the problem. This was of great importance since tho ! 

focus of tliis part of the paper and pencil exercises was on the decision issues 
rather than diagnosis. If left to their own diagnostic devices, it w'ould have 
i been unlikely that all pilots would have perceived the problem in tJie same way. 

j Next, the ramifications of the problem were clearly assessed. Having only 

battery power left to run electrical equipment, the problem was urgent in terms 
j of time. The consequences of flying beyond the lifetime of the battery w^ere 
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serious: the (light would be U'spped aloft w'lth no ineatis of communlcatloa 

or navigaUooal guidance. Finully, on esUmate of the degree of time urgency < 

i 

was given when the estimated maxlmun? time the ba.tery would be useable wxs - ^ 

stated as being not longer than minutes. ‘ ^ 

The paper and pencil scenario had now reached the point where the pilot \ 

was called upon to use Ms own personal decision skills iu the problem. The '' 

first task was to conduct an information search on the attributes of potential 
diversion airports. Tnls Included celling, visibility, nax^lds, terrain, 

availability of radar and distance and heading to the diversion airport. The ^ 

1 

purpose of this task was to determine tl\e search stratcg>' and decision rule * ‘ 

• • f 

the pilot used In shopping for an airport to which to divert . The second task 

involved ranking a group of sixteen airports from *’most preferable** to . " 

’least preferable” based on their attributes. * ^ 

\ 


The Information Seeking Task 

In this task the pilot was required to search for an airport to which to 
divert. The pilot was supplied with Figure HI dn Appendix H) which 
portrayed all the airports In the area, die was cautioned that all the airports 
shown should not be assumed to be within his range In terms of battery time. ) 
As he viewed the new' chart, the pilot was read the instructions given in 
Appendix H. The experimenter was t; act as the air traffic controller and 
would provide die pilot with the information he requested. The information the 
experimenter was prepared to give was summarized in Table A4 and shown to 
the pilot. 
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In aU there were sixteen potential diversion airports and tlj© experimenter 
had sl'c ^;iecea of Information (from four questions) on each. The total store 
of Info; matlon maintained by the experimenter Is given In Appendix H. 

E^ch pilot was told he had two minutes to conduct his search and to select 
an airport to which to divert. The mention of the two minute time limit was 
Intended to place a sense cf lime urgency on the problem, but It *vas not 
enforced. In most cases, however, the pilot had finished bis search and 
selected an alternate airport before the two minute limit had expired. 

The experimenter provided the pilot with each piece of information that the 
pilot requested. The experimenter recorded the sequence In v/hlch the 
Information was requested. The pilot continued to request Information 
until he had found an airport and revealed hls choice to the experimenter. 

At this point the iriformation seeking task was completed. 


F. The Ranking of Alternatives (Decision Phase) 

Information from ranking of alternatives was used to interpret die Informa- 
tion seeking phase. In this phase the pilot w'as asked to rank sixteen alternative 
airports from “most preferable” to ’least preferable” given hls problem situation. 
Each airport was described in terms of four attributes, namely, air traffic 
control (ATC) services at the aii*port, the v/eatlicr at the airport, the time to 
fly from present position to the airport, and th • best instrument approach 
facilities there. These attributes were chosen because they were Independent 
with respect to each other, and also because they were the more pertinent 
items to consider in this situation. 
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Each attribute was varied over two levels: a ”high** value (In terms of 
pilot preferability) and a ’low value”. (For example* weather 1000/3 vo. I 

500/l. ) All possible combinations of high and low attribute levels resulted 

i 

In a total of sixteen alternatives to be considered. The end product v/as a 

2"^ full-factorial design as shewn In Table VI-6. 

Each alternative airport was depicted on a 3 x 5 inch card in terms o£ 
the four attributes. The sbeteen cards v/ere shuffled (prior to the experiment) 
and laid out before the subject In u random fashion while the experimenter 
read these Instructions: 

”I have a set of cards here; each card describes an airport In 
terms of ATC services, weather, the flight lime from your 
present position to the airport, and the appioach facilities 
there. I would like you to rank these airports from your "most 
preferable" to ’least preferable", given the slbjatlon you are 
In. Recall that you have, at the very most, fifty minutes of 
battery time left. You may find It useful to divide the airports 
Into "subgroups", rank the airports in each subgroup, and then 
recorjiect tne subgroups as appropriate. Aftenvards, make a 
final check of your ranl< and adjust It as you think necessary. " 

Subjects w'ere given as much time as they needed to complete the ranl^lng 
task, but rarely did it take longer than five minutes. Wliile performing the 
task, subjects generally appeared quite involved and made meticulous 
adjustments to the rank before yielding a final ordering. When the subjects 
had completed the ranking task and were satisfied with their final product, 
the experimenter recorded the sequence and the ranking task was complete. 

In an attempt to estimate how "real" this simulation seemed to tlie pilot, 
and to determine the pilot’s relative risk taking tendencies, the experimenter 
posed a series of questions for the pilot to consider. The questions asked 
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Table VI-6. 2"* Fectoiial Layout of Airports 


i; 



Mrports 


Attributes 


ATC 

Weather 

7in:e 

Approach 

A 




4 - 

B 

- 

•f 

•f 

* 4 - 

C 

4. 

- 

+ 

4 - 

D 

- 


4 - 

4^ 

E 


•h 

- 

4 » 

F 

- 

-t- 

- 

4 - 

G 

-«• 

- 

- 

4 - 

H 

- 

- 

- 

4- 

I 

•f 

+ 

4 - 

- 

J 

- 

•f 

+ 

- 

K 

-f 

- 

4* 

- 

L 

- 

- 

4 - 

- 

M 

+ 

-f 

- 

- 

N 

- 

•f 

- 

- 

0 

P 

+ 

- 

- 

- 


(+ = High value; - = Low value) 

Key; ATC: + = Tower w/radar 

= UNICOM 

Time: + «= 15 minutes 

- = 30 minutes 

Weather: + *= 1000' ceiling, vis. 3 miles 

- = 500' ceiling, vis. 1 mile 


Approach: 


+ = ILS 
- •= NDB 
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the pilot how far he would go down hla list of ranked airports to find one 
with maintenance facilities to repair his alrplane--a question which seeks 
to find the limit of his diversion options. (The line of questioning used is 
contained under the Go-No-Go Instructions of Appendbt H. ) When the pilot 
finished these questions both the diagnosis and decision making sections of 
the paper and pencil exercises were completed, and the pilot was invited to 
participate in other events at the workshop. 


Analysis and Results 

The analysis proceeded first with the ranks provided by Uie subjects in 
the ranking task and then with the Information seeking data. The first part 
of the analysis was aimed at modeling die pilot's worth function and determin- 
ing if worth functions are related to pilot background variables. The theory 
of conjoint measurement was used to model the worth functions (Krantz and 
T\'ersky, 1971). The second part of the analysis centered upon the Informa- 
tion search and how it related to worth functions. The analysts which follows 
was performed on the first 20 subjects. 

Results of Ranking Task 

A list of the ranks made by tlie subjects Is given In Table \T-7. In this 
table the "name” of the airport refers to the airport with tlie same dimensional 
configuration as shown In Table \T-G. It should be mentioned that the airport 
name was not revealed to the pilot during the experiment in order to prevent 
any biasing effects that may have resulted. The numbers in Table \T-7 
correspond to the positions in the rank that the airports were assigned by the 
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subject. The conventloa was adopted that sixteen equals **most preferable”. 
It is evident that most of the pilots a^^eed airport A was most preferable 
and airport P was least preferable. However, much variation Is seen In 
the airports in between. 

The additive model in Equation (1) below was assumed to be the under- 
lying psychological process In the worth structures and w^as proven to be 
the correct choice through a series of axiomatic tests performed on the 
ranked data. (The ranked data of Subject 4 did not conform to the tests and 
his data was dropped from furtlicr analyses. In effect, the subject showed 
no logical preferred order. A 65 year old retired pilot, he mn.y not have 
understood the instructions. ) 


W(X,) = B,tc • ATC^ - • WX, . TIM^ + B^pp ' A PP^ (1) 

where WfX^) is the psychological W’orth of airport z, and ATC^., 'VX^, 
and APP;f th^ independent variables describing airport z In terms of 
ATC services, weather, time and approach aids respectively. The 


Independent variables took on a value of -^*1 for the high level or -1 for the 
low level of each attribute. The *TB”-coefflclents are the *Sveights” each 
subject assigned to a certain attribute In his ranking scheme. The B-coefflclents 
were obtained by performing a regression analysis where the ranlt position of 
airport z was substituted (according to conjoint measurement) for W(X^). The 
resulting coefficients are shown in Table VI-8. The range of values for the 
coefficients Is 0.250 to 4.000. An interpretation can be offered If one con- 
siders all four coefficients for each subject. The coefficients for Subject 1, 
for example, are 1.000, 2.000, 4.000, and 0.500 for ATC, weather, time, 
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Table VT-7. List c£ Airport Hanks by Subject 


;1 


' s • 


.1 


PC« AlRTCtni A TBUUC9 ft IA« WTf nUJtMUtlX. I* LCAITT fUlfrJUMLt 


1 

5 

A 

• 

C 

» 

£ 

P 

C 

B 

1 

J 

K 

L 

It 

R 

0 

f 

1 

lA 

14 

11 

IB 

B 

A 

4 

2 

IS 

13 

It 

<3 

7 

• 


1 

t 

14 

11 

IS 

1 1 

B 

a 

4 

a 

14 

lA 

IS 

4 

7 

4 

1 

1 

t 

lA 

11 

14 

11 

a 

T 

A 

B 


M 

IB 

4 

4 

1 

2 

1 

4 

lA 

14 

A 

1 

11 

1 

4 

IS 

12 

IB 

2 

i 

II 

4 

a 

7 

§ 

lA 

It 

11 

12 

II 

IB 

0 

A 

14 

T 

4 

3 

4 

1 

2 

1 

A 

lA 

11 

14 

11 

13 

(1 

IB 

4 

B 

7 

A 

1 

4 

a 

2 

1 

T 

lA 

T 

11 

A 

14 

4 

4 

2 

11 

B 

II 

1 

13 

3 

IO 

1 

• 

lA 

IS 

S 

A 

14 

13 

T 

1 

IS 

4 

4 

2 

II 

10 

3 

1 

4 

lA 

11 

If 

A 

14 

10 

IS 

2 

4 

B 

S 

3 

12 

? 

4 

1 

!• 

lA 

11 

1 1 

4 

IS 

12 

4 

3 

14 

0 

A 

2 

14 

7 

8 

1 

II 

lA 

B 

14 

T 

IS 

1 

13 

A 

12 

1 

10 

1 

11 

4 

4 

3 

11 

lA 

11 

11 

lA 

14 

II 

13 

4 

!• 

4 

7 

2 

4 

3 

S 

1 

tl 

lA 

11 

IS 

1 

T 

A 

4 

4 

13 

14 

II 

3 

iO 

B 

3 

1 

k« 

lA 

11 

B 

T 

14 

11 

A 

1 

4 

12 

4 

3 

13 

IO 

J 

1 

11 

lA 

14 

12 

lA 

A 

4 

T 

3 

11 

II 

11 

B 

4 

1 

6 

3 

lA 

lA 

14 

11 

lA 

a 

A 

4 

2 

IS 

13 

11 

4 

7 

A 

9 

1 

IT 

lA 

11 

14 

lA 

13 

II 
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Table VI- 8. Coefficients of the Additive Worth Function 


(SUBJECT 4 DEi-ETEB, 29 SUBJECTTS PEHAIBIKO 


OBJECT 

BSUBATC 

BSUBVX 

BSUBTin 

BSUDATP 

1 

1.000 

2.000 

4.000 

0.500 

2 

2.000 

0.750 

4.000 

0.625 

3 

o.noo 

1.000 

4.000 

2.000 

5 

l. 125 

2.375 

2.000 

2.675 

6 

0.500 

1.000 

2.000 

4.000 

7 

4. COO 

1.500 

1.500 

0.250 

8 

1.250 

4.000 

0.625 

1.623 

9 

2.500 

2.375 

0.625 

2.375 

le 

2.500 

3.375 

0.750 

1.075 

1 1 

4.000 

0.750 

0.625 

1.750 

12 

2.000 

0.750 

0.7,50 

4.000 

13 

1.625 

2.025 

2.500 

0.750 

14 

0.625 

4.000 

0.625 

1.750 

15 

1 . 875 

1.000 

3.075 

0.500 

16 

1 . COO 

2.000 

4.000 

0.500 

17 

2.375 

1.750 

2. 125 

2.375 

10 

3.250 

2.750 

1 . 375 

0.623 

19 

1.250 

1.250 

4.000 

0.500 

20 

I. 075 

1.000 

3.500 

1.375 

21 

2. COO 

4.000 

0.500 

1.000 

22 

3.500 

1.250 

0.750 

2.500 

23 

2.000 

4.000 

0.500 

I. 000 

24 

2.000 

0.875 

0.250 

4.000 

25 

1.000 

3.375 

2.000 

1.675 

26 

1.375 

1 .500 

0..500 

4.000 

27 

2.000 

1 . 000 

0.375 

4.000 

20 

0.1175 

4.000 

1.250 

1.375 

29 

0.875 

4.000 

2.000 

0,625 

30 

0.750 

4.000 

2.000 

0.750 
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and approach,* respectively. This can be Interpreted as follows: the 
worth of having the time value at the high level (15 minutes) was twice that 

of having the weather value at the high level (1000 feet ceiling and three miles 

S 

visibility), four times the worth of having ATC at the high level (tower with 
radar), and eight times the worth of having the approach at the high level 
(ILS). In other words, the most Important feature about each airport 
for Subject 1 was time followed by weather, ATC, and approach, respectively. 
In order to test for the validity of the additivity assumption In the model, 

the preference ranks determined from equation A for the sixteen airports 

i 

I were plotted against the original preferences for each subject. A Spearman 

' ranlc correlation was computed to estimate the fit of tlie model derived ranks 

I with the actual ranl^s. Figure VI-8 depicts a tjqjical plot. The correlation 

coefficients ranged from 8.74 to 1.00 for the 29 subjects Indicating the model 
additivity was an acceptable assumption. 

The next step was to determine if any relationship existed between the 
i worth function coefficients and pilot background variables. Since no measure 

of performance exists in this experiment, the data was examined to find 
i relationships or explain differences. The basic approach was to dichotomize 

I the sample population based on several different descriptors of a pilot’s 

background and skill. The meanr of the coefficients for the resulting two 
j grourii were then compared to see if any significant differences occurred as 

a result of the division. The divisions were performed on the basis of flight 
experience, training, of pilot certificate, t>Tpe of flying most commonly 

I 

I ^ done, and measures of ability determined by the knowledge survey and other 





w 


!W|fr* 


means. A summary of coefficient means by pilot category is provided In 
Table VT-9 along with the criteria used to split the sample population. 
Significant differences (at the p *< . 10 level) are enclosed la dashed lines. 

Analysis of Worth Functions 3y Subjects and Groups of Subjects 

The basic approach in this section Is to dichotomize the sample popula- 
tion based on several different descriptors of a pilot's background and skill. 

The splits are performed on the basis of flight experience, training, type 
of pilot certificates, type of flying most commonly engaged in, and measures 
of ability determined by the knowledge survey and other means. The worth 
coefficients of Table VI-9 become the center of attention in this analysis. 

This analysis covers the first thirty subjects used in this test. 

The first dichotomization is performed on the basis of total flight exper- 
ience in terms of flight hours. A bar graph representing the distribution of 
total hours is shown in Figure VI-9. As evident in this graph, the distribution 
of total hours is in no way ’’normal'*, and the criterion used to split the sample 
is somewhat arbitrary. However, the sample was split at the natural break 
nearest the 50th percentile. Because the sample is more heavily loaded 
with experienced pilots (many people consider pilots with 800 or more hours 
to be ’’experienced”) the search for the natural break in total flight time pro- 
ceeded from the mean toward the ’’inexperienced” end. The criterion used 
to split the pilots was 1100 hours. Nine pilots were in the lower category 
and 21 pilots in the higher category. A t-test was performed to determine 
If there were any significant differences between the means of the B coefficients 
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Table VI-9» Summary of Coefficient Means by Pilot Category 






Grcur> 

Coefficient M 

eans 



No. of 
Subjects 

Pilot Category 

B ^ 
ate 


'^tim 

B 

epp 


I 

II 

I 

11 

I 

11 

I 

II 

(22) 

(8) 

Total Flight Hours: 

Category I: Time > 1100 hrs. 

Category II: Time < 1100 hra. 

1.74 

1.90 

2.08 

2.54 

i.eo 

1.90 

1.91 

1.41 

(22) 

(8) 

Total Single Engine Hours: 
Category I: Time > SCO hra. 

Category II: Tine < 800 brs. 

1.91 

1.44 

2.05 

2.6S 

1.74 

2.04 

l.Sl 

1.41 

(lA) 

(16) 

IFR Hours 5 

•Category I: Tiae > 3C0 hrs. 

Category II: Tlase < 300 hrs. 

1.37 

1.94 

2.13 

2.28 

2.00 

1.68 

1.91 

1.70 

(7) 

(23) 

Type of Training: 

Category I: Hilltary Training 

Category II: Civil Training 

1 1.25 

-f 

1.92 , 

2.40 

2.16 

2,20 

1.72 

1.63 

l.SO 

(8) 

(22) 

Grade of Certificate: 

Category I: Airline Transport 

Category II: Private or CoL-mercial 

1.73 

1.80 

1 1.48 

2.49 I 

I 2.53 

1.56 1 

1.73 

1.83 


^Significant differences enclosed in dashed lines* 


Table VI-9 (con't. ) 






Group Coefficient 

Means 



Ko. of 
Subjects 

Pilot Category 

m 

®at 

c 

B 

wx 

®tlra 

B 

epp 



I 

II 

I 

II 

I 

II 

I 

II 

(5) 

(25) 

Type of Flying Mostly Dene: 
Category I; Military or Airline 
Category II: Pleasure, GA/Ccisii, 
Business 

r 

1 0.90 

1.96 1 

2.20 

2.21 

2.55 

1.67 

1.74 

1.90 

(17) 

(13) 

Knowledge Survey Score: 
Category I: Score > 12.5 

Category II: Score < 12.5 

1.86 

1.65 

2.06 

2.A2 

2.09 

1.45 

1.55 

2.08 

(16) 

(lA) 

Diagnostic Performance Score: 
Category I: Score > 13 

Category II: Score < 13 

1.77 

1.79 

2.25 

2.17 

1 2.25 

1.37 1 

1 1.37 

2.20 1 

(20) 

(10) 

Would Pilot Attempt Flight? 
Category I: Pilots said **y^8** 
Category II: Pilots said **no’* 

1.61 

2.15 

2.33 

1.96 

1.61 

2.31 

• 2.11 
1 

i.Ol I 

(15) 

(15) 

Airports Passed for Maintenance: 
Category I: A or more 

Category II: 3 or less 

1.63 

1.95 

2.60 

1.80 

I 1.40 

2.28 1 

1.96 

1.57 


*Slgnlf'''ant differences enclosed In dashed lines 


Figure VI-8. Distribution of Total Flight Hours of Sample Population 
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for the two groups. At the p " . 10 level, no significant differences were 
found. 

Splits of the sample were made based on the number of Instrument 
Flight Rules (IFR) hours and single engine airplane hours. A bar graph 
of the distributions of each are shown In Figure VI- 10 and VI- 11, 
respectively. As in the case of total flight hours, these distributions are 
far from nornal, and the "cut" was made in the came manner. At the . 10 
level, no significant differences were found. 

The t>"pe of training a pilot received was used as a criterion to split 
the sample. There were seven military trained pilots and twenty three civil 
trained pilots. A t-test was performed on the worth coefficients and a 
difference which was significant at the . 10 level (p = .06) wis observed for 
the mean value ol (Recall that Is a measure of the importance of 

air traffic control facilities in airport worth evaluation. ) For civil trained 
pilots the mean y.-Jlue of ivas 1.92 and for military trained pilots it 
was 1.25. 

There are several possible interpretations of this difference, but most 
allude to the pilot's attitude toward ATC facilities which are formed by pre- 
\ious exposure. In mlUtary pilot training programs, much more emphasis 
Is placed on emergency procedures and resolving In-flight problems than 
In civil pilot training. This may lead to an attitude of greater self-reliance 
in problem situations on the part of the military pilot, and a reluctance to let 
too much of the problem "out of the cockpit". Additionally, military trained 
pilots may feel more strongly that ATC facilities would be of only limited 
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Figure VI- 9 . Distribution of IFR Hours of Sample Population 
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Figure VI- 10. Dlslrlbutlon of Single Engine Hours of Sample Population 
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value In this scenario. Admittedly, If battery power had been depleted and 
no communications were possible, ATC services would be of no use at all. 

The pilot sample was split on the basis of the type of certificate the 
pilot held. In this case, the twenty one pilots with Private and Commercial 
certificates made up one group, and the eight pilots with Airline Transport 
Pilot certificates made up the other group. The split was made in this fashion 
because the differences between private pilots witii liistrumcnt ratings and 
commercial pilots are not great. The experience and proficiency require- 
ments for the Issuance of those certificates are nearly the same. In many 
ways, the commercial pilot training and certification process provides only 
slight extensions of skill to private pilots with Instrument ratings. On the 
other hand, the stringent eligibility and proficiency requirements for the 
issuance of the Airline Transport Pilot (ATP) certificate have led to the feeling 
that ATP airmen are the ’’cream of the crop”. The difference between the 
^vo groups of pilots In terms of tested ability Is distinct. 

Some notable differences were observed when comparing the worth 
cccfficients of these two groups. The mean value of a measure of the 

Importance of weather to a pilot In this situation, was 2.43 for private and 
commercial pilots and 1.48 for airline transport pilots. A t-test was performed 
and this difference was found to be significant (p = .05). Another difference, 
significant at the .10 level, was observed for the value of a measure of 

the importance of the time attribute. For airline transport pilots had a 
mean value of 2.53 and for private and commercial pilots the mean value was 
1.56. 
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The difference In 3^^^ between the two groups Is most likely a ftiuctlcn 
of training and the relative ’level of preparedness** to fly In adverse weather. 
Although pilots In both groups are trained in the procedures and maneuvers 
to be used when flyl3ig In bad weather, airline transport pilots are required 
to perform those maneuvers to much greater accuracy on flight tests. Also, 

In meeting the greater experience requirements for tl\e ATP certificate, 
airline transport pilots have been exposed to more poor weather situations 
tlian their private and commercial pilot counterparts. In summary, airline 
transport pilots have reason to feel more confident about their flying skills 
in relation to marginal weather. 

Some interesting comments can be made about the difference in mean 
values of for the two groups. Because the mean value of was 
higher for airline transport pilots (2.53) than for private and commercial 
pilots (1.56), one might casually suggest that airline transport pilots are 
more cautious. The higher value of time could be Interpreted as a desire 
of airline transport pilots to avoid flirting with the problem by landing quickly. 
This Is the opposite of what one would expect, especially in view of the 
supposedly ’'stress hardening*' experiences an airline transport pilot faces 
In kis training and career. However, a more probable explanation for the 
observed difference is that airline transport pilots can take advantage of their 
skills to consider more airports. They may feel better prepared to conduct 
flight operations into an airport which is nearby but has poorer weather, and 
can therefore take advantage of time. Private and commercial pilots, though, 
may feci that some airports, even though they are close by, arc beyond the 
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limits of their skills In terms of weather and facilities. KencOt they sacrifice 
time for better conditions and services. 

The type of flying most commonly done was also used as a basis to divide 
the pilot sample. Pilots who engaged primarily in business, light commercial, 
or pleasure flying made up one group. Pilots who were Involved with airline 
or military flying comprised the other gi oup. The split was made In this 
fashion because the highly structured environments !n which airline and 
military pilots operate are similar in many ways. They are both usually 
required to fly in and out of busy terminals and heed schedules, policies, 
and other disciplines. Pilots who fly for business, light commercial, or 
pleasure concerns, however, operate In a much more relaxed atmosphere 
and dictate their own policies. Based on this split of the sample population, 
a significant difference (p = .024) was observed for the coefficient 
The mean value for business, light commercial, and pleasure flyers was 
1.96, while the value for airline and military pilots was 0.90. 

An explanation of this difference could be that military and airline pilots 
fly much more frequently in congested areas, and are mostly under the sur- 
veillance of an ATC facility. Given Uils day- after-day exposure to ATC, they 
are more aware of its abilities and limitations. Another potential explanation 
is the same one noted earlier when contrasting civil trained and military trained 
pilots. Because of the intensive Initial and recurrent training in normal and 
emergency operations they receive, military and airline pilots may wish to 
solve in-flight problems with ’’on-board*' resources rather than let too much 
of the problem outside of the cockpit. 
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AH of the previous comparisons were based on classifications of pilot 
experience. There were, however, four measures of pilot ability and self- 
evaluation that were used to classify the pilots as well. The following dis- 
cussion treats comparisons made on the basis of the pilot’s Imowledge of 
aircraft systems, his ability to diagnose problems in flight, and estimates 
of the perceived risk he assigned to the problem. 

The knowledge survey which was administered to the subjects before the 
experiment was designed to estimate their knowledge of aircraft systems. 

The mean score was 12.3 out of a maximum of 20, and the distribution of 
scores was approximately normal, as seen In Figure VI- 12. Tlie division 
was made at the mean and no significant differences were found in the worth 
coefficients of the resulting two groups at the . 10 level. 

The pilot’s diagnostic ability was estimated in the first half of the decision 
making workshop. In the four diagnosis scenarios, pilots were scored on the 
closeness of their diagnoses to the real problems, and these scores were 
summed up to yield a total correct score. The spilt was made at the 50th 
percentile of the total correct score which ranged from 5 to 20. At the . 10 
level significant differences w'erc observed for two coefficients. Pilots in 
the lower half of the sample based on the total correct score had a mean value 
of 2.20 for the B^^pp coefficient, while pilots in the upper half had a mean 
value of 1.37 (B^pp is a measure of pilot worth for the instrument approach 
dimension, and, for this difference, p = .080). A possible interpretation of 
this difference Is that diagnostic ablllt>^ parallels a pilot’s perception of his 


Figure VI-11. Distribution of Knowledge Survey Scores 



scons: o.i 


flying skills. The ND3 approach, aside from being less accurate, requires 
more beadwork and skill than an ILS approach. 

The other coefficient in which a . 10 level significant difference was 
observed is (p ~ .076). Pilots in the upper half of the sample had a 
mean value of 2.25 while tliose in the lower half had a mean value of 1.37. 

This difference may again relate to the perceived level of skill. Pilots 
higher in diagnostic ability may not perceive the problem situation to be 
any more time critical than pilots in the lower half, but they can take 
advantage of closer airports more often. They believe they have the skills 
necessary to meet the challenges of poorer conditions which may accompany 
the closer airports. The notion that flying skills and diagnostic skills are 
related should be examined in future research. 

During the preflight preparation stage of the decision making scenario, 
each pilot-subject was asked if he would normally attempt the flight under the 
stated conditions. Ten of the pilots indicated they would not try it while 
20 said they would. A significant difference (p = .026) was observed between 
the two groups for the coefficient for approach aids, B^^pp. Pilots who said 
they would not attempt the flight had a lower mean value for B^^pp (1.01) than 
those who said they would try the flight (2. 13). This observed difference does 
not lend itself to a simple, straightforward interpretation. One would expect 
the relatively cautious pilots who would not attempt the flight to prefer the 
better approach aid (in tills case the ILS) so they could have more in their 
favor. Ho'wever, if one looks at the other differences in coefficient means 
even though they are not significant, some insight is gained. Pilots who would 
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not go on the flight had a higher value for time, and a lower value for weather. 

’ I This trend leads one to believe that the leery pilots do not wish to push their 

I luck In terms of time. Kence, they put more weight in the time factor and 
take emphasis away from other factors. 

i I 

j The final dichotomlzation of pilots was performed on the basis of their 

1 ^. responses to the questions on maintenance facilities, in essence, each 

pilot was asked how far down the list of airports, arranged from most pre- 
I ferable to least preferable, he would go to find the necessary maintenance 

facilities to repair his plane. Fifteen pilots said i, 2, or 3 airports and 
fifteen pilots said 4 or more (the range 1 to 14). The mean value of 


I was 2.28 for pilots v/ho responded with 3 or less (call them *’less risky*’ pilots) 

while the same measure for the (more risky) pilots who responded with 4 or 
more airports was 1.40. This was significant at die . 10 level (p = .083). 
j This difference can be attributed to conservatism of the pilots in the less ‘ 

risky group. In the same manner tliey are reluctant to take risks by "passing 
up" too many preferable airports, they are unwilling to pass over a closer 
j airport (in terms of time). 

A summary of the results of all the analyses performed in this section is 
given in Table VI- 10. It is interesting to note that significant differences In 
j worth function coefficients were not a result of flight differences, rather, were 

related to tho isrrade of pilut certificate., the amount and t^^ie of initial and 
recurrent training, and the type of flying most commonly done. This suggests 
that training and repeated exposure to testing situations are the variables which 
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can predict the general form of a pilot’s worth function. A closer examination 
of the training and certification process is in order. 

Once a pilot obtains a private pilot certificate with an Instrument rating, 
there is very little he must legally do to continue exercising the privileges of 
his certificate. He can continue to accrue many hours of flight time but he 
is required to demonstrate, on only a sporadic basis, that he is maintaining 
his basic skills. Airline transport pilots and those pilots who fly for the 
military or airlines, however, must maintain a higher level of skill regardless 
of the amount of flight time they have. Many are required to demonstrate 
proficiency in all sorts of demanding situations and at much more frequent 
Intervals than the biennial flight reviews required of general aviation pilots. 
The general ’’level of preparedness” is much higher for military and airline 
flyers than for the rest of the flying population. All of this lends support to 
the notion that the total amount of fll^.t experience is not as important as 
the amount and quality of initial and recurrent training in determining the 
general worth structure of a pilot. 

G. Results of Information Seeking Task 

Pilots were referred to the simplified charts of Appendix H when performin 
information searches. Because of the h^^pothesized strong winds aloft (out of 
the southwest at 30 kts. ), airports which were closer in terms of distance 
were not always closer in terms of time. Table VI- 10 shows rank orderings 
of the airports, from nearest to furthest, in terms of both time and distance. 
Table VI- 11 depicts a subject by airport listing of worth values. 


r 


Table VI- 10. Raiilc Orderings of Airports In 
Information Search From Nearest to Furthest 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 


In Terms of Time 
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A 
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L 

D 

C 

M 

G 

B 

K 


(12 minutes) 
(13) 

(18) 

(19) 

( 20 ) 

( 22 ) 

( 22 ) 

( 22 ) 

( 22 ) 

(23) 

(25) 

(26) 

(28) 

(28) 

(31) 

(32) 


I 

E 

O 

N 

P 

F 

H 

J 

G 

B 

A 

L 

C 

D 

K 
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(23 miles) 
(34) 

(40) 

(40) 

(41) 

(43) 

(49) 

(54) 

(54) 

(54) 

(58) 

(63) 

(63) 

(65) 

( 68 ) 

(69) 
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Table VI-11. Computed Worth Values and Means by Airports 
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All twenty-nine subjects considerer^ airport I (i.e., information was 
requested about I) and twenty-nine considered airports E and N. The 
frequency of consideration decreases for airports further away. The 
number of times each airport was chosen as an airport to which to divert 
in this scenario is given in Table VI- 12. The most popular choice was 
airport N. 

Each pilot’s search pattern was analysed in an attempt to determme the 
search and decision logic used when seeking information. Though many pilots 
displayed definite search patterns, determination of a decision rule was not 
possible for two reasons. The first is that many pilots reverted to standard 
ATC information exchange formats. In requesting weather information, for 
example, they would always request ^’ceiling” before ^Visibility” as in the 
format for such data in weather reports. U'he second reason is that many 
pilots estimated bearings and dis:ances directly from the chart, and hence 
they had information for which no record of request was made. 

Some interesting observations can be made, however, v/hen a comparison 
is made between the results of the ranking task and the infcrnxatiou search 
task. Table VI- 13 lists the pilots whose most important attributes were 
ATC, time, weather, and approach aids, respectively. 

Airports N and J were the only ones chosen in the group of pilots who 
considered ’time” to be the most important attribute. Airports N and J are 
in the top half of the airports ranked according to closeness in terms of time. 
Pilots who considered other attributes to be the most important, how’ever, 
chose a much broader range of airports. Pilots in the group v/ho considered 
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Table VI- 12. The Number of Times Each Airport 
Was Chosea as a Divers loo Airport 


Airport 


Number of Pilots 
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Table VI-13. Moat Important Attribute for Pilots 
j (Subject Numbers Shown) 
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ATC Weather 


Time Approach 
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S13 
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weather to be the most Important attribute chose airports A,C, J,M,N,0,P; 
pilots who thought ATC to be most Important chose airports 0,N,J,E, and 
pilots who thought Approach Aids were most Important chose airports 
A,C,J,N. The same trends were apparent in the airports considered by 
pilots In each group. The general Interpretation Is that pilots who placed 
most emphasis on "time" did not venhire as far to find a suitable airport 
as did pilots with other priorities. Pilots who placed emphasis on an attribute 
other than time, on the other hand, either were forced to search cont.'aually 
for airports with better conditions land maybe further away), or felt that 
time was not a serious issue and searches of airports further away were 
feasible. 

H. Analysis of P & P Tests (All Phases) 

Since the two major phases of the P & P tests yielded almost ninety 
measures on each subject (both raw data and derived measures) it became 
necessary to develop an overall analysis strategy to derive the maximum 
amount of useful Information from such data. Note that the data was divided 
Into four basic categories: The Diagnostic Phase Data, the Decision Phase 
Data, the Subject Biographical Data and Knowledge Data. Those data bases 
were individually analyzed for descriptive statistics and derived measures of 
performance. 

In the diagnostic phase the raw data Included (for each scenario and for all 
scenarios) merit, correctness score, efficiency, total Inquiries, total unique 
tracks, total tracks, criticality assessments before and after the diagnosis, use 
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o( control inputs and derived measures o Z (correct score/ total trades) end 
COHINQT (correct score/total Inquiries). These variables are defined In 
tlie glossary. 

For the diagnostic phase, means and variances of these measwes both 
for each subject across the four scenarios and for all subject across each 
rcenarto were reported In the Master Data Table (Table VI- 14). In addition, 
the distribution of these data was examined for outliers. Rank correlations 
were examined to find associative relationships (see Table VI- 15). These 
data were then used In tlie combined data analysis w^hlch Is described below. 

The decision phase data Included subject airport rankings, information 
seeking profiles, go/no go responses before the flight and the number of 
airports the subject w*as willing to pass to locate an alrcralt mechanic. 

As described above, conjoint measurement analysis was employed on the 
ranking data to derive worth functions. These were expressed as pilot 
weightings for weather (WX), navigation aids (APP), radar services (ATC), 
and '“•me to the airport (TIM) (see Glossary). The decision phase data were 
then used in the combined analysis. 

'i ;ie third major data base involved subject biographical and knowledge 
data. These are shown in Table VI -14. These data were subjected to descrip- 
tive iMutistlcal analysis and transformations to adjust for outliers In distri- 
butions. Table VT-14 describes the findings from the knowledge and biographical 
data. 

The main thrust wav *Jie combined analysis of diagnostic data, decision 
phase data, and knowledge and biographical data. Sections H, J, and K which 
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Table VI -14. Master Data Table 
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Table VI-14. Master Data Table (con*t.) 
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Table VI-14. Master Data Table (con't.) 
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Table VI-14. Master Data Table (con’t. ) 
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follow detail the results of this combined analysis. Easentlally three 
types of analysis were Involved. First, Spearman Rank Correlations were 
examined for all data Inputs with empniasls on those correlations with major 
dependent measures. These included merit, efficiency, and correctness 
measures and their derivatives from the diagnostic phase and weighting 
functions from the conjoint measurement analyols of the decision phase. 

The rank correlations provided not only Insight Luo two variable relations 
but indicated input variables for the stepwise regressloa analysis which 
follow’ed. 

The stepwise regressloa analysis predicted dcpendcat measures from 
the set of independent measures (biographical, knowledge measures and 
Independent performance measures). 

Finally, partitions on the independent measures were examined to ascer- 
tain differences in performance measures. The splits \’aried depending on 
the nature of the variables in question, e.g. , pilots with aircraft mechanic 
(A and P) ratings vs. pilots without A and P differences In pilot 

ratings etc. In addition, some performance measures such as total merit 
were split Into top and bottom quartiles to ascertain differences in other per- 
formance indices. The performance measures are listed and defined In the 
Glossary, Table VI-3. 



\ ‘V 


I. A Rank Correlation Analysis - Combined Data 

Table VI- 15 depicts the Spearman rank order correlations for the major 
Independent and dependent variables In the analysis. Th^se correlation tests 
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were conducted as a ftrat cut through the data prior to stepwise regression. 
The correlations enabled the researchers to get an overall view of relation- 
ships between the variables cf Interest. Clearly, some 8ignlx*cant correla- 
tions resulted because the t^vo variables were related not each other but 
through a third variables. These effects were evident with the stepwise 
ro^n*osslon. 

Table VT-15 reports rank correlations for experience variables, knowledge 
scores and overall tests. Although a log transform of total and single engine 
hours was employed to adjust for the skewed distribution In these factors, 
such a transformation has no effect on rank correlation. The comments to 
follow consider a relationship to be significant If the a value is 10. 

It Is Interesting to note that total hours Is related only to single engine 
hours and not to any of the performance measures. Slagle engine hours is 
related to knowledge scores. This is to be expected since the knowledge 
test and Its subscores were based on single engine aircraft operation. The 
negative correlation of the decision factor, weather, with single engine hours 
suggests that high experience levels lead to less emphasis on weather in the 
diversion decision. 

Knowledge scores are highly related to total merit (a *= .0001) - a finding 
that holds up In the entire analysis. Knowledge about aircraft subsystems 
definitely affects diagnostic performance In a positive manner. Knowledge Is 
also related to total correctness ( a *= .005) and total efficiency ( a * .003) 
both of which make up total merit scores. Hence, the knowdedgeabla pilot Is 
more likely to not only get the right answers In diagnostic tests but also to get 
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Table VI- 15. Rank Correlation Table 
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Table Vl-15. Rank Correlation Table (con*t.) 
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O.OOG3 

0.01 14 

40 

40 

40 

40 

40 

40 

43 

40 

49 

49 


TIM CATSeni 
0.37473 0.36730 

tt.oilia 0.0197 
39 40 


TOTCOU IXITAL COIUIECT SCORE 


TOTCOR 

TOTMEUIT 

COR I ROT 

rr 

TOTEFF 

KNOI^TEDG 

CATSCR3 TOlTRiXKS 

TOTIKa 

CATSCH2 

1.00000 

e.ar>(i77 

0. a 1024 

o.Boini 

C. 45644 

e. 43005 

0.39aOU -0.311323 

-0.36202 

0.30923 

0.0000 

0.0001 

0.0001 

0.0001 

0 . 003 1 

6.0056 

0.0119 

0.(il46 

0.C2I4 

0.0223 

40 

40 

40 

40 

40 

40 

40 

43 

49 

40 


CATSCUI TIM 

0.27305 0.27123 

0.0302 0.0949 

40 39 


TOir.FF 


TOTAL t '»^IENGY SCORE 


TOIYFF 

IXITINQ •. 

JUT 

COIUHOT 

ZT 

ixnruAKs 

TX)TC:OH KN01.TEDC 

CAi’sciia 

CATCCR! 

1.00000 

-0.30490 

02il4 

0.76446 

0.69330 

-0 . 67337 

0.45644 

0.45493 

0.45462 

0.33733 

0.0000 

0.0001 

O.UOOl 

0.0001 

O.OCOi 

0.0001 

0.0631 

0.0032 

0.0032 

0.0332 

40 

40 

40 

40 

40 

40 

40 

40 

40 

40 


CWnU.lXiT TIM 

-0.29990 0.211733 

0.0601 0.0739 


Table VI- 15. Rank Correlation Table (con't.) 


DACKCROUin). DIACROSIS. ARD TOTALS 

6PEAWIAR COIUUXATIOH COEFf ICIERTS ✓ Pft03 > IRI UMCEit Ea:IUJa*® / ItU^SER OF OBSEaVATIC.TS 


con! HOT 


cimihOT 

mrmrjiiT 

ZT 

TOTCOri 

TOTTFP 

TOTir^Ci 

TOTTHAKB 

kkovi.kim; 

CA1KCJX2 

CATECm 


0.93193 

0.90234 

0. a 11)24 

0.76446 

-0.75639 

-0.6331)6 

0.03917 

0.47356 

0.36967 

o.oce4) 

0.0001 

O.OUOl 

0.0001 

0.0001 

e. 4)001 

0.0001 

0.0603 

0.6620 

0.6109 

43 

40 

40 

40 

40 

40 

40 

40 

40 

40 

TIM 

CAi>;r.ui 









0.353f.U 

0.34397 









0.0372 

0.02311 









U9 

7T 

40 









7T 

coiuKUT Trmo.MiT 

TOTTZUKS 

TOTCOa 

TOTEFF 

TDTIMO 

KKOVIT-DC 

CATsnna 

CATSCHI 

1.00000 

0.V0234 

0.1)3702 

-0.C3027 

e.Boisi 

0.69330 

-6.63662 

0.07330 

0.40534 

0.42991 

0.0000 

0.0001 

0.0001 

0.0001 

e.oaol 

0.6001 

6.0001 

0.0001 

e.ooio 

0.0670 

40 

40 

40 

40 

40 

40 

40 

40 

40 

49 

CATSUIT 

TIM 









0.33974 

o.cajca 









0.0226 

o.oaoi 









40 

39 









riK 

TIME 









TIM 

CATSCrUl 

jcnovixDC 

APP 

Ttm3:niT 

COlUHQT 

TOT! no C.TITXTCrr 

CATPCr^2 

ATC 

i.eoooo 

0.41)210 

0.43533 

-O.3E026 

0.37473 

0.35363 

-0.32076 

-0.31774 

0.31763 

-0.32 1C6 

0 • OOGO 

0.0019 

0.0036 

e.0169 

o.oiaa 

0.6272 

e 0465 

0.04??7 

0.0403 

0.0533 

39 

39 

39 

39 

39 

39 

39 

39 

39 

39 

TO nyy 

7.T 









0.21)733 

0.21)361) 









0.0739 

O.OIIOI 









39 

39 









LPP 

AITUOACn 








APP 

Till 

WX 

CATscnn 

KJfOWI.FDC 

TOTTILM^S 

SF.nK3l/JG 

TUTina 

ATC 

CKTOLTOT 

1 .CODUO 

-0.31)026 

-0.37606 

-0.26993 

-0. 17361 

-0. 13176 

0. 14413 

-0. 13137 

-0. 129)2 

0. 11069 

0.0000 

0.0169 

0.011)3 

0.0963 

0.2a49 

0.3564 

0.3314 

0.4233 

0.4334 

0.4034 

39 

39 

39 

39 

39 

39 

39 

39 

39 

39 

Ton:oii 

CATS('R2 









-0. 1 1332 

-0. 10497 









0.4043 

0.3241) 









39 

39 









IX 

KEATlIhR 









wx 

API* 

1 

i 

ATC 

TIM 

Ainroirre 

TIinSILOC 

TOTIHQ 

TOTCOa 

coaiwoT 

i .00000 

-0.37606 

-0.21)740 

-0.22632 

-0.22126 

6.21019 

-0. 19035 

0. 11931) 

-0. 1C991 

-0.O934T 

0.0000 

o.oio:) 

0.0761 

O. 1653 

e. 175(1 

0. ia33 

0.226& 

0.4692 

0.OD54 

0.0714 

39 

39 

39 

.39 

39 

39 

39 

39 

39 

39 


Tabic VI-15, Rank Correlation Table (con't.) 


vx 

TOTm'IUT 

-0.OVO03 

0.636a 

39 

ATC 

ATC 
1 .OOUOO 
0.0000 
39 

AIT 
•0. 12912 
0.4334 
39 

TOTTIUKS 

V-* 

TornuKs 

1.00000 

0.0000 

43 

AIRTOinS 

0.33719 

0.0330 

39 

TOT 1 m 

TOTIWQ 

1.00003 

0.0000 

40 

CNTia.TOT 

0.31692 

0.0463 

40 

crmiLioT 

cnmiTOT 
I. 00000 
0.0000 
40 

TornL\Ks 
0.22303 
0. 1650 
40 







BACKCnOUT^D. DIAGNOSIS, AI?D TOTALS 


SPEAUItVn ConilELATION COEFFICimS / PROD > IKI UNDER llOxFOIO*© ✓ ITUnUER OF OBSERVATIONS 

VF>.\THFJ1 


TOTTRAKS 

0.07193 

0.6632 

39 




AIR mUTIC CONTROL 


TIM 

CNTRI/rOT 

TOTINQ 

VX 

TOTTRAKS 

CATSCrUJ 

ZT 

COniKQT SEHRSLOC 

-0.31 IR6 

0.29194 

0.23332 -0.22652 

0.20651 

-©. I934U 

-e. 16400 

-0. 1393a 

0. 13713 

0.0533 

0.G713 

0. 1529 

0. 1655 

0.2072 


0.31D5 

9.3323 

0.4250 

39 

KNOWLKOG 
-0. 103.15 
0.5305 
39 

39 

39 

39 

39 

a© 

39 

39 

39 


I 

3 


TOTAL ^ OF TRACKS I 

7T TOTFKF TOTINQ CORINQT TOTTIZRIT KKOWLEDC CATSCU2 CATSCRl TOTCOR | 

-0.03027 -0.67337 0.66629 -0.653Q6 -0.56 I2U -0.43033 -0.42539 -0. 39553 -0.33323 i 

0.0001 0.0001 O.UOOI 0.0001 e.OG02 0.0029 0.0061 0.0120 O.0E46 J 

40 40 40 40 40 40 40 49 40 ! 

TIM I 

-0.22li03 
0.1627 
39 


TOTAL ^ OF INQUIRIES 



com NOT 

TO nUAKS TOTMERIT 

ZT KNOKLEDC 

CATSCR2 

Tt)TCOR 

TIM 

-0...^ rOO 

-0.75639 

0.66629 -0.64515 

-0.63662 -0.46343 

-0.45030 

-0.36232 

-0.32076 

o.ooot 

0.0001 

0.0001 

0.0601 

0.0001 0.6023 

0.G036 

0.0214 

o.o:c5 

40 

CATb'ClU 

-0.304,10 

0.0561 

40 

40 

40 

40 

40 43 

40 

40 

39 


CATSCIU KNOVI.FDC 

TIM 

TOTINQ 

TOTEFF 

A^TC 

IXITTIERIT 

CORINQT 

ZT 

-0.44103 -0.35232 

-0.31774 

0.31692 

-O.2V990 

0.29194 

-0.23020 

-0.24919 

-0.23727 

0.0044 0.0250 

0.0407 

0.0463 

9.0601 

0.0713 

G. 1976 

e. 1210 

0. 1404 

40 40 

39 

40 

40 

39 

40 

40 

40 

CAT>»(:U2 

-0.21244 








0. 1002 








40 
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Tabic VI- 15 


Rank Correlation Tabic (con*t.) 


BAOCCaOUITD. DIAC7V0SIS, AJRD TOTALS 

SPEAIIMAJI COrUlFXATIOR COEFFICIEKTS / PllOIS > IRI UWDER O^snilO*® / irOT^ZR CF CCS SllVATlOi^S 


AiRTOirrs 

TOTiKQ Tin vx Tnn::Loc zr cntt.ltw 

e. 245134 -0,23210 0.21519 -0.2C943 -0,13337 0.13164 

9.1274 0.1349 0.1003 0.2306 0.4CI3 €.<244 

39 39 39 39 39 39 

CATSCK2 Kauwi.k.im; 
e. 1 1021 -9. io:h9 
0.3042 0.3307 

39 39 


AIWOHTS 
I .OOOCO 
0.0000 
39 


TOmUKS 
e. 33719 
0.0330 
39 


CATSr.RI 

0.3lO;i7 

e.0403 

39 


TfmrfF 

0.2B3O3 

0.9706 

39 


to 

00 







1 


f 



the answers more efilclently, l.e. he uses few Inquiries to get the answer. 

As might be expected, and CORINQT are also highly correlated to know- 
ledge scores since they are derived from total correctness. 

One major finding In the correlations (w’hlch holds up with subsequent 
analysis) is the positive relation of knov/ledge with time (TIM) (a .0056). Tills 
means that more knowledgeable pilots place high emphasis on *’tlme to alr- 
port*' in the case of a destination diversion decision. With concern tor 
possible additional complications, knowledgeable pilots want to get the 
aircraft on the ground at the earliest opportunity rather than proceed farther 
to better weather or facilities. 

As expected, total correctness Is related to Z^, CORII\^QT, total merit, 
and individual scenario scores. Table VI-15 also Indicates pilots who have 
high correctness scores use fewer tracks and fewer inquiries than those with 
lower scores (TOTTRKS, a = .0146) (TOTINQ, a ^ .0214). This again 
supports the link between diagnostic correctness and efficiency of diagnosis. 

Use of control activation related inquiries w^as negatively related to 
knowledge ( a = .0258), to total efficiency ( a = .0601) and to time weights. 

This was somewhat surprising In light of the value of selected control input 
tests to find problem causes, e.g. , prop cycling to ascertain the locus of low 
oil pressure readings in scenario 1. 

The decision weights show obvious negative intercorrelations with each 
other since one cannot have high w'eighting on more than one attribute. The 
time-knowledge relation described above is again noted. Airports passed 
to locate mechanics in the decision tests is negatively related to efficiency 
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and knowledge scores and positively to total tracks used. If airports passed 
represents a crude measure of risk then high risk acceptance pilots have 
inefficient diagnostic procedures and are less knowledgeable. 


A 


J. Regression Analysis 

In order to ascertain what factors predict performance measures^ a 
series of stepwise regressions were performed. These are listed In 
Table VI-16. The table Indicates the dependent variable, the n, and 
the significant predictor variables from three sets of independent variables - 
biographical and experience, knowledge, and other independent performance 
measures. Candidate predictor variables are indicated at the top of the 
table. The general strategy was to Include all predictor (independent) 
variables, even those which could be related to each other, e.g. , DIFT 
(total tracks - total unique tracks) and total tracks and total unique tracks. 

The model then selected which of these added the most to the prediction. 

The model permitted no variable to be Introduced If it had been derived from 
another significant pred ctor variable. 

On the other hand, no predictor variable which formed part of the 
dependent variable was allowed in the regression. For example, since 

= total correct/total tracks, a predictor variable could not be Introduced 
which contained total tracks, e.g. , DIFT (DIFT = total tracks - total unique 
tracks). 

WTien correctness scores on individual scenarios are examined, C1-C4, 
some Interesting results anpear. Efficiency is a good predictor of correctness 
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for scenarios 2, 3, and 4. Knowledge subscores are good predictors for 
Cl and C4 only. Experience variables (training and rating) show up in Cl 
prediction and (recency) in C3. These are far less significant than efficiency 
as predictors. Control use is clearly Important in scenario 3 as might be 
expected (this was the rough running engine scenario). The number of 
inquiries per track is a positive predictor of correctness in scenarios 
2, 3, and 4. It seems v^vident that scenario 1 differs from the other three 
scenarios probably because of its unique nature, i.e. , it required the 
pilot to seek Information about conditions Inside the cockpit, but not on the 
panel. Weather weighting was a negative predic .or for C2 and C4. 

Pilots with good total correctness scores are characterized by high 
efficiency, a low number of total inquiries, a low number of tracks, a 
low DIET (total tracks - total unique tracks) and thus a high number of 
inquiries per track. Knowledge is also a good predictor. Note, no 
biographical and experience factor predicts total correctness. 

Individual scenario efficiency scores show little predictability from 
knowledge test scores. Efficiency on scenario 1 Is related to experience 
variables, I.e., training, rating, and recency - again supporting earlier 
findings on Cl. Most of the efficiency predictors on the scenarios include 
total tracks (negatively) for E 2 » ^ 2 * E^, and unique tracks for E 3 . Interest- 
ingly enough, control activation is a good predictor (negative) for El, E2, 
and E4. 

For total efficiency prediction, total tracks, and total control movements 
are negatively significant. Total correctness is also a significant predictor, 

again Unking good performance with efficient performance. 
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Tabic VI- 16 


o 

Final Summary of Stepwise Predictions 


PRFDICTOR VARIABLE 

niOGnAPHlCAL & EXPERIENCE 

KNOWLEDGE 

INDEPENDENT PERFORMANCE VARIABLE 

o 

0 

o 

o 

N 

o 

Agc-S.E.IIrs.-Tot Hrs.- 
Recency- Flying- Rating- 
Training 

CATSCRl 

CATSCR2 

CATSCR3 

KNOWLEDG 

C1-C4, E1-E4, UT1-UT4, TT1-TT4, INPTRl- 
INPTR4, CB1-CB4, CNTRL1-CNTRL4, DIFl- 
DIF4, TIM, APP, WX. ATC, AIRPORTS. 
TCRITBEF, TOTEFF, CNTRLTOT. TOTINQ, 
TOTUTRKS, DIFT, INPTRT 

Cl 

J oo 

.49 

39 

Training (.0329)* 
Rating (.0004)* 

CATSCR3 (.0053) 

CNTRLl (.07C8)* 
DIFl (.0827)* 

C2 

.40 

39 



WX (.0041)* UT2 (-0366) 

AIRPORTS (.0978) IN1>TR2 (.0015) 

E2 (.0002) 

C3 

I-* 

’ C3 

to 

.70 

39 

Recency (.0019)* 


WX (.0458)* CNTRL3 (.0010) 

CBS (.0114)* INPTR3 (.0013)* 

E3 (.0001) 

o C4 

.00 

39 


CATSCRl (.0289) 
CATSCR2 (.0124) 

E4(.0001) INPTR4 (.0385) 

UT4 (.0001) 

' TcrrcoR 

o 

.50 

39 


KNOV/LEDG (.0231) 

TOTEFF (.0636) DIFT (.0057)* 

TOTINQ (.0928)* INPTRT (.0225) 

TOTTRAKS (.0034)* 

El 

o 

.74 

39 ’ 

Training (.0219)* 
Recency (.0209) 
Rating (.0091)* 


TTl (.0004)* CNTRLl {.0001)+ 

CB1 (.0551) DIFl (.0581) 

K2 

.63 

39 


CATSCR2 (.0034) 

CNTRL2 (.0007)* 
TT2 (.0078)* 

E3 

.81 

39 



APP (.0023)* UTS (.0001)* 

ATC (.0070)* C3 (.0001) 

tM 

0. 

.81 

39 

Rating (.0040)* 


TIM (.0330) CNTRL4 (.0544)* 

TT4 (.0001)* C4(.0016) 


♦ Negative B-Value 


Tabic VI- 16 (con’t.) 


^ PHKDICTOR VARIABLE 

BIOGRAPHICAL & EXPERIENCE 

KNOWLEDGE 

INDEPENDENT PERFORMANCE VARIABLE 

■b 

^ TOTEFE 

r2 

.GO 

N 

39 



CNTRLTOT (.0782)* TOFCOR (.0306) , 

TOTTRAKS (.0001)* 

o TOTMERIT 

.5G 

39 


KNOWLEDG (.0001) 

DIET (.0022)* 

> ZT 

.G1 

39 


KNOWLEDG (.0121) 

TOTEFF (.0317) 
TOTUTRKS (.0290)* 

CORINQT 

.55 

39 


KNOWLEDG (.0006) 

DIFT (.0006)* 

CNTRLTOT 

.40 

39 

SEIIRSLOG (.0784) 
Recency (.0062) 


TIM (.0087)* 
TOTUTRKS (.0097) 

TIM 

.4G 

39 


CATSCRl (.0353)* 
CATSCR2 (.0001) 

AIRPORTS (.0027)* 

. ATC 

.20 

39 

Training (.0793) 
Age (.0093) 



M \VX 

IS 

.35 

39 

TIIRSLOG (.0246)* 
Training (.0020)* 


AIRPORTS (.0359) 
TOTUTRKS (.0418)* 

AIRPORTS 

.54 

39 


CATSCRl (.0555)* 
CATSCR2 (.0023) 

WX (.0905) 
DIFT (.9001) 

KNOWLEDG 

.G2 

39 

Flying (.0023)* 
Ti’ainlng (.0052) 
Recency (. 0292)* 


APP (.0042)* 

TOTTRAKS (.0314)* j 

■ . ^ -■ , — ■ 


♦ Negative B-Value 


, I fi 


-c 


i 


In general, efficiency measures led to be prediction than correctness 




scores (higher R squares). What was surprising was the absence of know- 
ledge, biographical and experience variables as significant predictors for total efficiency. 

For total merit prediction, the knowledge score is the best predictor 
( a = .0001). Later, when extreme scores on total merit are examined through 
t-tests, knowledge scores differentiate w^ell betv/eea high and low merit scores. 

Total merit is also predicted by DIFT. Low DIFT yield high merit scores which 
suggests a nonrandom approach by high performance score pilots. Not sur- 


prising, and CORINQT were much like total merit since all three measures 


are functions of total correctness. Knowledge v/as a significant predictor for 
these performance measures. No experience variables were significant. 
CORINQT was like merit. revealed total unique tracks and total efficiency 
as predictors. 

Table VI- IG also shows that knowledge scores are significant predictors of 
time weighting and airports passed in the decision phase. In both cases 
CATSCRl was a negative prediction. CATSCRl deals with engine and fuel 
systems. CATSCR2 (electrical and cockpit operations) appears to be more 
germaine to the destination diversion decision. As expected, pilots, who weight 
time as critical, pass few airports. Pilots who consider weather weightings 
as critical are willing to pass airports to locate a mechanic. Although the 
R squares were low, biographical and experience factors appear for weather 
and ATC weighting. These will be examined later in partition testing. 

Since knowledge has bearing on performance it was decided to seek pre- 
dictors of the total knowledge scores. Table VI- 16 reveals that type of flying. 
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training, and recency of flight all are significant oredlotors. Non-pleasure 
flying, high recent experience and military training yield higher knowledge 
scores. Both approach aid weightings and total tracks are good predictors 
of knowledge in an inverse direction. 

K. Tests on Data Partitions 

Table VI- 17 shows a series of tests on extreme partitions of major 
independent and key dependent measures to ascertain if differences might 
exist in extreme cuts through the data as compared to regression of the entire 
data set. Test candidates for the partitions included twent^'-one performance 
variables. Two state variables are easier to examine in this framework, 
i.e., mechanics vs. non-mechanics, military vs. civilian training and 
go vs. no-go preferences for the decision flight. Tv\'enty dependent 
measures shown in Table \^-16, plus two experience measures were considered 
for performance differences in the data splits. 

Pilots with recent flight experience (over fifty hours in the past 3 '^ear) 
were more knowledgeable and used less control activations than pilots with 
less than Uventy hours in the past year. Surprisingly enough, diagnostic 
and decision performance were not different - perhaps due to the small 
number of low recency pilots. 

Total hours of experience ( < 1007 hrs. vs. > 5375 hrs. ) showed no 
relation to performance. Separating single engine hours experience revealed 
greater knowledge for pilots with over 2075 single engine hours vs. those 
with less th^m 488 hours. Splits o) ratings revealed commercial and ATP 
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rated pilots have more knowledge than private pilots. For training 
partitions, military training led to higher knowledge scores than non- 
military training. A split on EFR hours revealed a a!i{^t total correctness 
difference with the two groups ( < 175 hrs. vs. > 700 hrs.). In all of the 
splits above there was little or no performance difference. 

When pilots under age thirty are comt>arcd to those over fifty* performance 
dllferences begin to appear. Younger pilots have higher merit, and CORINQT 
scores than older pilots, yet have less experience (total hrs. and single engine 
hrs. ). T\pe of flying also showed performance effects. Pleasure flying 
pilots showed less knowledge, less experience, less efficiency, more tracks 
and larger DIFT than pilots who fly for airline, commercial, business, or 
military purposes. 

WTien performance measures are split to get profiles of high score pilots 
vs. low score pilots, many other independent performance effects are noted. 
Focus is on dependent variables or other measures not related to the parti- 
tloner variable. When partitioning on knowledge, the high knowledge scores 

(216) are associated with: 

a) higher single engine hours experience 

b) higher weightings on time in the decision phase 

c) higher correctness and merit scores 

d) higher efficiency scores 

e) fewer Inquiries 

f ) fewer tracks 

g) fewer unique tracks 

than pilots with lov/ knowledge scores dess than 10). 
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TaMc VI- 17 
Summary of T-Tesls 


DATA PARTITIONS ON 

% UUKIRAPIIICAL AND KXPERIENCE 

AND KEY DEPENDENT MEASURES 

N 

TEST VARIABLES - CATSCHi (.0809) (0) means rcccncy differences on CATSCHl 
was significant at = .0863. The (0) means high recency group had higher mean - 
values than low rcccncy. 



CATSCRl, CA'rSCn 2 . CATSena. KNOWLFDG. TOTMFRrr. TOTEFF. 
TOTCOH. ZT, CORINirr. ATC, TIM. APP. WX. PROPCOrTT. TOTJNQ, 
TarUTRKS, TCHTRAKS, TIIRSIXXJ, SFHRSI.OG, TDELTAC, DIPT i 

INPTRT 

Hiy;h Recency vs. Low Heccney 
c (0)> r>0 hrs. in past yr. 

(3) <20 hrs. in past >T. 

o 

(0) -25 

(1) -6 

CATSCRl (.08GD) (0) 

CATSCR3 (. 0020) (0) | 

KNOWKEDG (.0110 (0) 1 

PROPCONT (.0170) (1) 1 

SEHRSLOG (.0040) lO) 

►-» Low Total Hrs. vs. High Total Hrs. 

(0) < 1007 lol*)l hrs. 

(1) > 5375 total hrs. 

(O)-IO 

(D-lO 

SEHRSLOG (.0001) (1) 

Low Single Engine Hrs. vs. High Single Engine Hr: 
(C)^ '188. V5 single hrs. 

(1)> 2075.25 single lirs. 

(0) -IO 

(1) -io 

CATSCRl (.0011) <1) 
KNOV.T.EDG (.0073) (1) 
TlIRSLOG (.0001) (1) 

» Privule Hating vs. Non- Private Hating 

(0) • private rating 

(1) - commercial rating and 

air tr.'uispor! rating 

(0)-3 

(D-31 

CATSCR2 (.0770) (1) 
KNOWI.FDG (.0185) (1) 
TlIRSl.OG (.0004) (1) 
SEMRSI.OG (.0400) (1) 

YOUNG vs. OLD 

(0) ^ :io >Ts. old 

(1) > 50 yrs. old 

0 

o 

0 

o ^ 

(0) -IO 

(1) -)1 

CATSCU2 (.0059) (0) 
TCrrANCRIT (.0889) (0) 
ZT (.0288) (0) 
CORINQT (.0C43) (0) 
ATC (.0335) (I) 
TUnSLOG (.0140) (1) 
SEIinSI.OG (.0198) (1) 
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Tabic VI-17 (con't.) 


• ! 


DATA PAim riONS ON 
HDXIRAPIIICAL AND ICXPFRIKNCK 
ANDKKY DKPKNDKNT MKASURLS 

N 

1 rST VARIABLFS 


MKCn vs. NON-MKCn 


CATSCRl (.0035) (1) 

O 

(0) N=35 

(0)-35 



<1) N"5 

(l)-5 



>.!lUlary v3. Civilian 


CATSCR2 (.0001) (2) 

O 

(1) MlUlary 

(1>-10 

KKOWLELK; (.0108) (2) 

o 

(2) Civilian 

(2)- .'10 

T11R.SI.OG (.0000) (1) 


Pleasure vs. Non- Pleasure 


CATSCRl (.0008) (0) 


(0) airline 

(0)-31 

TOTEFF (.0090) lO) 


commercial 

(D-9 

TOTTHAKS (.0300) (1) 


business 


TIIRSI.OG (.0001) (0) 

W 

military 


SEIIRSLOG (.0210) (0) 

C3 

o>- 

(] ) pleasure 


DIFT (.0122) (1) 

0 

Low IFR llrfi. vs. High IFR lire. 


TOTCOR (.0815) (0) 


(0) < 175 IFR hra. 

(0)-9 

THRSLOG (.0001) (1) 


(1) > 700 IFR hrs. 

(l)-9 

SEIIR.SLOG (.0180) (1) 

I.OW 

Knowledge Score vs. High Knowledge Score 


CATSCRl (.0001) (1) 


(0) £i 9 on knowledge Ic&t 

(0)-9 

CATSCR2 (.0014) (1) 

0 

(1) > IG on knowledge lest 

(D-8 

CATSCR3 (.0001) (1) 

o 

O 

,0 



TOTMERIT (.0070) (1) 

0^. 

0 



TOTEFF (.0103) (1) 

oo 



TOTCOR (.0430) (1) 
ZT (.0038) (1) 
CORINQT (.0004) (1) 
TIM (.0734) (1) 

o 

o 



TOTINQ (.0059) (0) 
TOTUTRKS (.00.59) (0) 
TOTRAKS (.0310) (0) 

o 



SEIlRSLa: (.0127) (1) 




DIFT (.0519) (0) 
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Tabic VM7 (con't.) 
Summary of T-Tcsls 


DATA PAirrmONS on 
UKX inAPmCAl- AND KXPKRIKNCE 
AND KEY MKASUHES 


N 


TEST VARIADI-ES 


GO vs. NO GO 
( 0 ) 

(1) 


(0) -9 

(1) -30 


ATC (.0790) (0) 
APP (.0081) O) 




Lew Total Merit vs. High Total Merit 

(0) ^129.25 TOTMERIT 

(1) ^235.00 TCrrMERIT 


(0) -IO 

(1) -IO 


w 

<o 


CATSCRl (.0124) (1) 
CATSCR2 (.0022) (1) 
CATSCR3 (.0010) (1) 
KNOWLEDG (.0001) (1) 
TOTEFF (.0001) (1) 
TOTCOR (.0001) (1) 

ZT (.0001) (1) 
COniNQT (.0001) (1) 
TIM (.OOGl) (1) 

TariNQ (.0012) (0) 

TOTIJTRKS (.0199) (0) 
TOTTRAKS (.0010) (0) 
DIFT (.0005) (0) 


Lo"' Total Correct vs. High Total Correct 

(0) ' 10 total correct 

(1) ^17 total correct 


(0) -ll 

(1) -ll 


CATSCRl (.0202) (1) 
CATSCR2 (.0183) (1) 
CATSCR3 (.0150) (1) 
KNOWLEDG (.0009) (1) 
TOTMERIT (.0001) (1) 
TOTEFF (.0024) (1) 

ZT (.0001) (1) 
CORINQT (.0001) (1) 


TIM (.0479) (1) 
TOTINQ (.0281) (0) 
TOTliTRKS (.0999) (0) 
TOTTRAKS (.0125) (0) 

SEiinsLcx; (.0752) (i) 

DIFT (.0097) (0) 
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Table VI- 17 (con't.) 
Summary of T-Tcsls 


° DATA PAKTmONS ON 

nUX^RAPniCAL AND EXPERIENCE 
AND KEY MEASURES 

N 

TEST VA.RIABLES 

o 

Low Total Efficiency vs. Hij;h Total Efficiency 
total efficiency 
(l)-50 toUil efficiency 

o 

0 

(0) -IO 

(1) -a 

CATSCni (.0325) (1) TOTINQ (.0001) (0) 

CATSCR2 (.0090) (1) TOTIJTRKS (.0008) (0) 

KNOWLEEX3 (.0032) (3) TOTTRAKS (.0003) (0) 

TOTMERIT (.0003) (3) DIFT (.0007) (0) , 

TOTCOR (.0028) (1) ^ 1 

ZT (.0001) (1) ' i 

CORIKQT (.0001) (1) ] 

PROPCONT (.0985) (1) j 

Low Time vs. }Ii[;h Time 

(0)' . G25 time 

U)-2. time 
o ' ’ 

o 

(0) -13 

(1) -16 

CATSCR2 (.0706) (1) \ 

CATlSCRn (.0035) (1) ' 

KNOWLKOG (.0221) (1) j. 

TOl’MERIT (.0074) (1) i’ 

TOTEFF (.0125) (1) i| 

TOTCOR (.0494) (1) j| 

ZT (.0110) (3) 1 

COniNQT (.0127) (1) II 

TOTINQ (.01.30) (0) || 

TOTTRAKS (.0.365) (0) || 

IJIFT (.0043) (0) 1 

Mechanic Uelp Available 

T\vo or liCbs AlrporUi passed vs. more tlian tw( 

(0) < 3 alrixn ts 

(1) > 2 airports 

0 

‘o 

1 

(0) -ll 

(1) -28 

CATSCRl (.0802) (0) | 

TOTEFF (.0220) (0) -I 

TOTINQ (.0693) O) \ 

TOTUTRICS (.0828) (1) 

TOrTRAKS (.0163) (1) 

DIFT (.0310) (1) 

» • •-. # . a 1 ►•“.'1 W 
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When the top and bottom quartlle on merit and correctness scores are 
examined, an Interesting pattern emerges. I/f embers ox the upper quartlle 
ot performance scores are associated with: 

a) more knowledge 

b) greater weights or time 

c) fewer Inquiries 

d) fewer tracks 

e) fewer unique tracks 

f ) lower DIFT 

g) higher cfflciencj’ 

h) higher single engine hours (total correctness only and here 

o = .07) 

WTien upper and lower quartiles in total efficiency are examined much the 
same patterns as indicated above result. 

Since time weighting appears to be associated with good diagnostic per- 
formance, splits on time weights were made to test for performance effects. 
High time weightings look again like the list above, l.e. , greater knowledge, 
efficiency, correctness and merit with few^er tracks, fewer inquiries and low^er 
DIFT. 

In the regression analysis, airports passed was predicted by knowledge 
subscores, time weighting, and DIFT. Table VI- 17 shows that a split on 
airports (2 ^ vs. > 2) shows that low number of airports passed is related 
to hi^er efficiency and knowledge and lower values for DIFT, unique and total 
tracks and total inquiries. 

Ml 
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L. GAT Subjects in P/P Tests 


Eight of the subjects who participated In the FMS experiments also 
participated In the P/P experiments. All of the subjects from the first 
GAT scenario (fuel loss) and all from the second GAT scenario (partial 
power failure) participated. None from the third GAT scenario (partial 
navigation system failure) participated. 

Subjects 11, 33, 34, and 35 were used in the fuel loss FMS experiment. 
Subjects 28, 31, 32, and 38 were used in the partial power loss FMS exper- 
iment. 

3ome topical performance measures for this group of subjects are high- 
lighted in Table VI- IS. The encouraging thing about this comparison is that 
the results appear to be reasonably consistent. The subjects for each FMS 
experiment are equally distributed with respect to airman certificates held 
(one Pvt. , two comm. , and one ATP in each group) in similar portions to 
the entire P/P gn:oup. 

Each GAT subject was ranked from one (best performance) to four 
(poorest performance) by a subjective evaluation of the experimenter present 
during all of the GAT runs. Each subject was ranked separately for ’’Aviating", 
"Navigating", and "Communicating" for the scenario In which he participated. 
"Aviating" reflected basic stick and rudder skills. "Navigating" reflected the 
ablllt}’ to follow the flight planned course and the subjects awareness of location 
along the route. "Communicating" reflected the professional nature of the 
information exchange between pilot and controller. All GAT subjects took 
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T 'ble VI-18. GAT Subjects P/P Performance 


Subject 

GAT FMS 

Airman 

Rating 

GAT Perfor- 
mance Rank* 
A N C 

Open Knowledge 
Survey Score 
GATKT 

Closed Knowledge 
Survey Score 
KNOWLEDGE 

TOTCOR 

TOTMERIT 

11 

1 

2 

2 

2 

2 

5.02 

13 

10 

107 

33 

1 

3 

1 

1 

1 

5.63 

13 

12 

164 

34 

1 

2 

4 

4 

4 

4.39 

9 

7 

117 

35 

1 

1 

3 

3 

3 

3.12 

7 

15 

172 

28 

2 

1 

3 

4 

4 

3.79 

10 

15 

ICO 

31 

2 

3 

2 

2 

3 

5.5G 

16 

15 

245 

32 

2 

2 

1 

1 

1 

5.00 

15 

18 

236 

38 

2 

2 

4 

3 

2 

4.59 

7 

2 

10 


♦A = Aviating 
N = Navigating 
C = Communicating 


the open form knowledge survey prior to the experiment. These data are 
also noted In Table VI- 18. 

The first observation to make is that knowledge scores on both forms of 
knowledge survey seem to agree. Those scoring high on the open-form also 
score high on tlie closed-form. 

The second observation to malte is that GAT performance rank is generally 
consistent with total correct (TOTCOK) and total merit (TOTMERIT) scores 
on tlie p/p scenarios. In GAT scenario 2 the rank order of ’’Aviating** in 
exactly the same as tlie rank order of TOTMERIT. Other rankings are less 
perfect but still exhibit the same general trends. 

Although the sample size is too small to draw definitive conclusions, taken 
as a group, results for these eight subjects seem to Indicate that either FMS 
or p/p scenario experiments can be used to evaluate pilots with some 
assurance that relative rankings will be preserved. 

M . Summary 

The above analyses have found the same pattern running through the data. 
The three statistical approaches lead to the same conclusions. These are: 

1. ) Correctness in diagnostic scores is highly related to efficiency 

in reaching diagnostic answers. 

2. ) Most biographical and experience variables do not appear to be 

related to diagnostic performance measures (the exception is the 

A & P mechanic rating which does appear to be related to 


diagnostic performance). 






3. ) Knowledge scores are positively related to both correctness and 
efficiency in diagnostic performance. 

4. ) Of the four airport attribute weightings only high time weighting 
is related to high diagnostic performance. 

5. ) Patterns of information seeking for high diagnostic performance, 
l.e., high efficiency and high correctness, involve 

a) a minimal number of tracks employed 

b) a minimal number of unique tracks employed 

c) a minimal number of total inquiries 

d) a small number of track repeats, i.e. , DIFT (total tracks - 
total unique tracks) is small. 

Although most experience and biographical factors failed to be significant 
in the analysis, some did appear to be related to knov/ledge scores - such as 
high total single engine hours, high recency, military training background and 
non-pleasure flying. 
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VII. ANALYSIS OF PROCEDURAL COMPLIANCE 


A. Background 

At the same time that the GAT and paper and pencil scenarios were 
being developed for single pilot CIFE*s, a parallel study directed toward 
airline cockpit crew operations was undertaken. The person responsible for 
this effort was Lt. Col. Jeffrey Schofield who performed the research as 
his Ph.D. dissertation project. A copy of that dissertation, **Alrcrew 
Compliance With Standard Operating Procedures As A Component of Airline 
Safety*’, Is on file as part of this project’s records at NASA- Ames. 

Schofield used data generated in an experiment conducted in 1976 by 
Dr. H. P. Ruffell Smith under the auspices of the NASA-Ames Research Center. 
The Ruffell Smith research utilized a full-mission simulation to study the per- 
formance of fully qualified airline crews under varying conditions of ’vorkload. 
The cockpit was that of a Boeing 747 which accomodated the usual three-person 
crew plus two observers, a simulator operator /traffic controller, and an 
audio coordinator. The full-mission scenario used was built around a charter 
flight from Dulles Airport to Heathrow Airport (London) with a thirty-minute 
Intermediate stop at Kennedy Airport (New York) for fuel and cargo. The first 
segment placed relatively low workload on the crews, while the second seg- 
ment was much higher due to pre-programmed mechanical failures. 

Rufi^ll Smith concentrated on crew errors during tiie second segment (high 
workload) of the scenario. He was interested in establishing statistically 
significant physiological or historical predictors of crew performance during 
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the second leg. Schofield, on the other hand, chose to emphasize the routine 
or customary tasks of flight operations as exemplified by the first segment 
of the Ruffell Smith scenario. Furthermore, he was concerned with: 

1) quantifying routine crew procedures, 2) analyzing observed crew errors 
to identify which particular crew members were tlie prlm.ary causes of such 
errors, and 3) comparing measures of procedural compliance and operator 
error. 

The primary data used by Schofield came from tie audio tracks of the 
FM tapes and handwritten documents generated by the Ruffell Smith study. 

This information was supplemented by data which was culled from the Aircraft 
Operating Manual, the Company Operations Manual, the Federal Aviation Regu- 
lations, crew handbooks, and assorted navigational documents. 

B. Procedures 

A procedure Is defined as ”a symbolic and mnemonic representation of a 
set of sensory, cognitive, and/or motor activities which, when recalled and 
executed within determinable tolerances, complete a task as designed”. 

The word ’’procedure” and its many aliases appear throughout aviation litera- 
ture. Schofield identifies nineteen separate words and phrases associated with 
aircrew operations which have procedural conotation. 

Schofield enumerates a set of normal operating procedures, as opposed to 
Abnormal, Alternate, Irregular, or Emergency procedures, which represent 
an idealized sequence based on the events in the Dullcs-JFK segment of the 
Ruffell Smith experimental scenario. All of his procedures are considered 


mandatory for normal flight operations In insti^ument meteorological conditions. 
Each procedure is Identified by published format and tlie cockpit cr^w mem- 
bers expected to exhibit active procedural behavior. These are catalogued 
in Table \TI-1. 

The astonishing fact in this list is that 97 normal operating procedures 
can be identified for standard cockpit activities lasting approximately 75 
minutes. This lengthy list does not Include any ^optional” procedures or 
emergency type procedures. They represent only standard operating proced- 
ures for the first leg of the simulated flight scenario. 

Schofield has identified several empirical taxonomies which seek to 
classify these procedures in ways to identify useful relationships among 
them. One such grouping is tlie set of 21 crew cc ”>rdinatlon procedures shown 
in Table VII-2. Crew coordination procedures are emphasized since the\' cap- 
ture the essential ingredients of group leadership, crew management and 
behavioral conformity. Schofield examines the realtlonships between meticu- 
lous compliance with coordination procedures and the crew errors noted by 
Ruffell Smith. 

Compliance Assessment 

Although Ruffell Smith used eighteen crews in his experiment, the quality 
of the data generated and the observers were not the same for all eighteen 
simulation runs. Schofield selected ten runs, which had the same set of 
observers and usable audio data throughout, for detailed procedural analysis. 
The 21 crew coordination procedures were further subdivided into checklists. 


Table VH-1 


NORMAL OPERATING PROCEDURES 


Index 

Number 

Name 

Format^^^Cperator^^^ 
codes codes 

1. 

Basic ATC communications practices 

N 

PNF 

2. 

Preflight Radio Checklist 

C,N 

P2 

3. 

Gear pin status report 

H 

PI ® 

4. 

Hydraulic system pressurization 

N 

PI & FE 

5. 

AXIS report 

N 

U 

6. 

Clearance Delivery communications 

N 

PNF 

7. 

Ground Control communications 

N 

PNF 

8. 

Pre-start Checklist 

C,N 

A 

9. 

Ground crew report 

H 

PI 

10. 

Cabin report 

N 

PI 

11. 

Engineer's Start Checklist 

C,N 

FE 

12. 

Start Checklist 

C,N 

A 

13. 

Engine starting 

N 

PI & FE 

14. 

Ground connections and hand signals 




report 

N 

PI 

15. 

Engineer's Taxi Cliecklist 

C,N 

FE 

16. 

Pre-taxi Checklist 

C,N 

A 

17. 

Transfer of EGT monitor 

N 

PI & FE 

18. 

Ground Control communications 

N 

PNF 

19. 

Taxi 

N 

PF 

20. 

Takeoff and departure briefing 

H 

PF 

21. 

Final weight and balance computation 

G,N 

FE 

22. 

Taxi Checklist 

C,N 

A 

23. 

Tower communications 

N 

POT 

24. 

Passenger pre-takeoff announcement 

N 

PF 

25. 

Engineer’s Takeoff Checklist 

C,G,N 

FE 

26. 

Runway line up 

N 

PF 

27. 

Takeoff Checklist 

C,N 

A 

28. 

Thrust setting (takeoff power) 

G,N 

PF & FE 

29. 

Takeoff 

N 

PF 

30. 

Takeoff callouts 

N 

POT 

31. 

Noise abatement departure 

G,N 

PF 

32. 

Gear retraction 

N 

PF & POT 

33. 

Departure Control communications 




(initial contact) 

N 

PNF 

34. 

Thrust setting (rated power) 

G,N 

PF 6 FE 

35. 

Departure Control communication 




(radar vector) 

N 

POT 

36. 

Flap retraction 

G,N 

PF & POT 

37. 

Altitude callout 

N 

PNF 

38. 

Intermediate level off 

N 

PF 

39. 

Departure Control communications 




(climb clearance) 

N 

PNF 


..>r 
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Table VTl-1 (con’t.) 


rr) T2) 


Index 

Nuaber 

Noae 

Format 

codes 

Operator 

codes 

40. 

AirwAyt navigatloQi practices 

N 

?F 6 PNF 

41. 

Thrust setting (rated power) 

G,N 

F & FE 

42. 

C.liab (below 10,000 fret MSL) 

N 

PF 

43. 

Coupany departure report 

N 

U 

44. 

AKTCC cccsunlcat ions (initial contact) 

N 

FNF 

45. 

Scat belt *igu 

N 

U 

46. 

Cllnb (above 10,000 feet KSl) 

}l 

?F 

47. 

After Takeoff CheckJLlst 

C,N 

PF S Ft 

48. 

Altloetez reset (not applicable for 
cruising below 18,000 feet) 

H 

A 

49. 

AJITCC coczaunicacions (route clearance) 

N 

PKF 

50. 

Cruise data 

C,H 

FE 

51. 

Altitude callout 

N 

PNF 

52. 

Level off 

N 

PF 

53. 

Hach nuober/alrspced crosscheck 

H 

FE 

54. 

Cruise 

H 

PF 

55. 

*\RTCC coaaunlcatlons (radar vector) 

U 

FNF 

56. 

Turbulence penetration 

N 

A 

57. 

ARTCC cocsuoicat ions (radar vector) 

N 

PKF 

58. 

Turbulence exit 

N 

A 

59. 

AKTCC cotsDunicacions (route clearance) 

N 

PNF 

60. 

Fuel syatens canagerient 

C,N 

FE 

61. 

AKTCC cocTaunicaticns (center change; 
initial contact) 

N 

PKF 

62. 

ATIS repo4f 

N 

U 

63. 

Coapaay arrival report 

U 

u 

64. 

Approach briefing 

K 

?r 

65. 

AKTCC conrsuni cat ions (sector change; 




Initial contact) 

N 

PhT 

66. 

ApprcMch data and speed bugs 

C,N 

A 

67. 

Passenger arrival announcciaent 

H 

PF 

68. 

Descent Checklist 

C.N 

A 

69. 

AKTCC ‘oscLunicat Ions (descent clearance) 

N 

P\T 

70. 

Descent (above 10,000 feet MSL) 

U 

Pr 

71. 

Alticecer reset 

N 

A 

72. 

Seat belt sign and landing lights 

N 

U 

73. 

Descent (below 10,000 feet HSL) 

H 

PF 

74. 

Approach Control communications 
(initial contact; clearance) 

K 


75. 

Approach Checklist 

C.}i 

A 

76. 

Category I Instrument L.'nding System 
(ILS) Approach 

C.N 

PF 

77. 

Approach radio checks 

N 

?F & PNT 

- o 
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Table VH-1 (con’t.) 


Index 

Nuaber 

Hacne 

Formatf^^ Operator^ 
codes codes 

73. 

Altitude callout 

N 

PNF 

79. 

No smoking oign 

N 

U 

80. 

Approach Control cocraunications 
(radar v€ctor) 

N 

PNF 

81. 

Approach flap extension 


PF & PNF 

82. 

Course bar and glide slope callouts 

N 

PNF 

83. 

Approach Control comunications 
(approach clearance) 

W 

' PNF 

84. 

Landing gcar/landing flap extenaloa 


PF i PNF 

85. 

Landing Checklist 

C,N 

A 

86. 

Final approach fix (FAF) coriDtjoicat Ions 

H 

FNF 

87. 

FAT InstruDcnt crosscheck 

N 

PNF 

88. 

Precision approach callout 

H 

PKF 

89. 

Outside scan and visibility callouts 

ll 

PNF 

90. 

Landing 

H 

FF 

91. 

Landing roll callouts 

N 

PNF & FF 

92. 

Tower coemunicat ions 

N 

PNF 

93. 

After Landing Checklist 

C»K 

A 

94. 

Taxi 

H 

PF 

95. 

Ground Control cocaunicat ions 

W 

PNF 

96. 

Parking 

N 

PF 

97. 

Blocks Checklist 

C,N 

A 


(1) Format Codes: C (Checklist), G (Graphical), N (Narrative) 

(2) Operator Codes: A (All), PI (Captain), PZ (Copilot), FE (Flight Engineer), 

PF (Pilot Flying), PNF (Pilot Not Flying), U (Unspectried) 
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Tablo Vn-2 


CREV CX)ORI)l NATION PROCEDURES 
(To be used for quancitative coccpllsncc asscssocncs) 


Index 

Letter 

A. 

B. 

C. 

D. 

E. 

F. 

C. 

H. 

I. 

J. 

K. 

L. 

M. 

K. 

0 . 

P. 

Q. 

R. 

s. 

T. 

U. 


Procedure Knrae 

Pre-start Checklist 

Start Checklist 

Prc-tsjcl Checklist 

Transfer of EOT Monitor 

Taxi Checklist 

Takeoff Checklist 

Takeoff Callouts 

Gear Retraction 

Flap Retraction 

Altitude Callout 

After Takeoff Checklist 

Altitude Callout 

Transfer of Aircraft Control 

Descent Checklist 

Approach Checklist 

Altitude Callout 

Approach Flap Extension 

Landing Cear/Landlng Flap Extension 

Landing Checklist 

Precision Approach Callouts 

Landing Roll Callouts 


callouta, configuration changes > and transfers. Performance of each of the 
ten c:*ew8 was then evaluated for each subdivision. 

Pre-start, Start, Pre-Taxi, and Takeoff Checklists are supposed to be 
Initiated upon command of the ^aptaln or the flying pilot. The other pilot Is 
then (o announce the name of tlie checklist as a confirmation of the command, 
and read the opening challenge. Once Initiated, checklists may be delayed 
by interruptions, but ultimately must be resumed and completed In toto. 

In every experimental run the requisite challenges and responses were 
made, even though some of the operator actions and replies were contrary 
to procedural specifications. However, there were remarkable differences 
In the patterns of behavior noted among crews for those five checklists. In 
a total of fifty opportunities over ten flights, the comm.and-aunouncement-challenge 
sequence was fully executed only live times. Hie observed shortcuts raised 
questions in Schofield’s mind about possible degradation In crew cohesion 
loading to increased uncertainty and lack of Internal order. 

The five audible checklists conducted by the two pilot crewmembers, were 
contrasted with three checklist sequences (Descent, Approach, and Landing) 
in which the flight engineer was the challenger. Exactly half of the observed 
thirty sequences here began In the prescribed command-announcement-challcngo 
order and only one was missing the Initial command. In addition to collectively 
making more of the prescribed announcements than their pilo* counterparts, 
the flight engineers were more self-consistent. Tliree engineers omitted all 
announcements and three othci'S omitted one. They also were more consistent 
than pilots in fo 'owing the response to the last challenge statement with the 


1 1 If fia < jiippy 

I 

i 

I 

prescribed procedure completion statement. When a pilot was the last 
challenger, 20% of the time the completion statement was omitted; when an 

^ engineer was the last challenger, only 4% were omitted. Schofield hypothesizes 

that crew coordination might be Improved by making the flight engineer the 
challenger of all checklists. 

Callout procedures are fundamentally different from checklists. In the 
usual format the non-flying pilot acts as a back-up or second-level visual 
monitor who audibly relays operating information to the flying pilot. Callouts 
occur during take-off, climb, descent, approach, and landing. 

Schofield identifies 170 opportunities, among the ten crews, to execute 
callout procedures. Thirty-eight pi'ocedural errors were noted, half of 
which were errors in altitude callouts during climb or descent. Tlie errors 
noted were callouts made by the flying pilot rather than the non-flying pilot 
(seven cases), late callouts (thirteen cases), and omitted callouts (twenty cases). 

Procedures for gear and flap extension/rctraction were well executed In 
terms of established oral procedures. In 104 observed configuration changes 
one of the two prescribed verbalizations was omitted four times, and one 
change (from flaps 1 to flaps up) was made without comment from either pilot. 
However, Schofield noted that aircraft altitude and location over the ground 
varied considerably at the Initiation point of selected configuration change 
procedures (e.g., the Noise Abatement Departure Procedure), which were to 
be performed simultaneously. 
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Verbal Indicators of the transfer of Exhaust Gas Temperature (EGT) 

Monitor and Transfer of Aircraft Control Procedures typify the quality of 
communications between specific pairs of crew members. In only two of 
the ten simulated flights does the flight engineer fail to advise the flying 
pilot when he can relinquish responsibility for monitoring EGT. However, 
in spite of obvious needs to effect the optional transfer of control procedure, 
t%vo crews never use it and three crews execute incomplete double transfers. 

Only one crew uses more than tvi’o transfers (4) duririg the simulated flight. 

Schofield further develops the thesis that verbal behaviors dictated by 
the aforementioned crew coordination procedures can reasonably be expected 
to enhance crew-coordlnatlon and flight safety. He also notes that non-compliance 
appears to d.pend more upon the operators involved than on the requirements 
of the procedures. 

D. Errors and P>rocednral Compliance 

Schofield modified and expanded the Ruffell Smith error counts so that 
every error is identified and individually related to an operator or ^oup of 
operators. Those data are summarized In Table VTI-3. The error categories 
coded by responsible operator are: pilot flying (PF), pilot not flying (PNF), 
captain, co-pilot, pilot team, flight engineer (FE), and entire crew. These 
categories cover all the errors recorded. 

The next step was to investigate potential relationships benveen tlie 
enumeratlve error data and the enumeratlve procedure compliance data. 

Because of the limited sample size, relationships noted below should be taken 
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Table VII-3 


RUFFELL SIXTH’S ERRORS® ATTRIBUTED TO OPERATORS 


Operator(o) 

Code 



Experioental Run Number 




3 

4 

5 

6^ 

8 

10 

12^ 

IB** 

14 

15 

Pilot flying 

PF 

4 

1 

1 

X 

6 

1 

0 

1 

4 

0 

1 

Captain 

CAP 

4 

1 

1 

6 

2 

0 

3 

2 

0 

1 

Pilot not flying 

PNF 


1 

0 

6 

2 

2 

3 

2 

4 

0 

Copilot 

COP 

1 

1 

0 

6 

2 

2 

X 

4 

4 

0 

Pilot team 

PTM 

1 

2 

2 

4 

3 

2 

2 

4 

1 

2 

Flight engineer 

FE 

0 

0 

2 

0 

0 

0 

0 

0 

0 

0 

Entire crew 

CRW 

1 

0 

2 

3 

1 

1 

1 

1 

1 

1 


Error cnCet^ories and descriptlcns are individually related to operators in 
Appendix D« 

^Oq runs 6, 12, and 13 the captain ia the PKF and the copilot is the PF. 
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as Indications of fruitful directions for furtlier research rather than as 
definitive results. 

A set of fifteen dependent variable categories (error counts) was generated 
by creating various combinations of six of tlie categories noted In Table VTI-3. 

A set of seven independent variables (five involving procedural compliance and 
two involving crew experience) was also generated as noted In Table VII-4. 
Stepwise multiple regression techniques were then used to identify the best 
models relating the Independent (procedural) variables to each of the dependent 
(error) variables in tiirn. Results of that analysis, noting Independent variables 
included and the maximum coefficients of determination, are shown in Table 
vn-5. 

Dependent variables, which reflect errors by the flying pilot (PF, TPF, 
CPF), by the captain (CAP, TCAP, CCAP) and by the two pilots collectlvciy 
and individually (PLTs), all have highly significant regression models in 
which pilot flying checklist commands (PFCK) and non-flying pilot callouts 
(PNFC) are the common independent variables. That is, pilot errors do 
appear to be related to those two classes of procedural non-compliance. 

Procedures Summary 

The Schofield study of procedural compliance by aircrews who participated 
in the Ruifell Smith experiment suggests the following observations: 

1) Crew members face an impossible challenge in attempting 
to mentally catalog all of the standard operating procedures 
(SOP) published for them. 
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2) Routine noa-complianco with an assortment of SOP’s has been 
documented. 

3) Forty-five percent of the enumerated crew errors Involved 
two or more operators, which suggests that human redundancy 
by itself does not erradicate personnel error. 

4) A statistical link appears to exist beb'^^een operator errors 
and procedural compliance. 

5) Full mission simulation offers nev/ possibilities for studying 
aircrew behavior In a controlled, high fidelity, operational 
setting. 

6) Altitude callouts, which duplicate functions performed by a 
machine, produced the highest frequency of non-compliant 
behavior, suggesting that Uiey may need modification. 

7) Lack of unitary leadership and internal coordination was most 
often observed when the captain was not flying the aircraft, 
suggesting a need to redefine flying co-pilot responsibilities. 


A 

4it» 


F. Procedures Epilogue 

As a follow-on to the Schofield research, a current flight engineer for one 
of the major carriers (who is also a graduate student at OSU) was invited to 
critique the study and to suggest a method for scaling criticality of normal 
operating procedures. The conclusion of the critique was that the Schofield 
research was a valuable first step in supplementing the standard human-engineering 
approach used in aircraft accident investigations. Such invesiigatlons often 
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Tabic VII-4 


INDEPENDENT VARIABLES 







Experimental Run Number 




Variable name 

Code 

J 

4 

5 

6 

8 

10 

12 

13 

16 

15 

PF checklist coinnands 

PFcac 

5 

5 

8 

1 

3 

7 

3 

3 

5 

6 

Pilot checklist announcements 

PA 

2 

2 

1 

5 

0 

1 

0 

0 

2 

2 

FE checklist announcements 

FEA 

3 

3 


2 

0 

2 

1 

0 

1 

0 

PNF callouts 

PNFC 

11 

13 

13 

9 

15 

16 

15 

13 

15 

14 

Aircraft control transfers 

. TRAN 

3 

2 

k 

0 

3 

6 

C 

2 

4 

4 

Crev tnenbers vdth more than 
1000 hours in B-7A7 

CREX 

3. 

2 

2 

1 

2 

3 

. 1 

1 

2 

• 2 

Pilots with more than 
1000 hours in B-747 

PEX 

2 

1 

1 

1 

2 

2 

0 

0 

1 

1 


Table VII-5 


KAXIl-IUM COEFFICIENTS OF DETERMINATION 


Dependent 

variable 



Independent variables 


One 2 
Code r 

Two 

Codes 


Three 

Codes 


PF 

PNFC 

.749 

PFCK, PNFC 

.869 

PFCK, FEA, PNFC 

.912 

CAP 

PNFC 

.651 

PFCK, PNFC 

.824 

PNFC, TVJii, PEX 

.878 

PNF 

PFCK 

.579 

PFCK, PA 

.709 

PFCK, PA, PNFC 

.864 

COP 

PFCK 

.513 

PFCK, TRAN 

.642 

PFCK, ?A, TRAN 

.738 

PTH 

CR£X 

.446 

CREX, PEX 

.626 

TRAN, CREX, PEX 

.665 

TPF 

PNFC 

.571 

FEA, PJIFC 

.785 

PFCK, FEA, TRAN 

.843 

TCAP 

PFCK 

.595 

PFCK, PCTC 

.825 

PFCK, FEA, PNFC 

.856 

TPHF 

PFCK 

.692 

PFCK, PA 

.763 

PFCK, PA, PNFC 

.808 

TCC? 

PFCK 

;567 

PFCK, TR7M 

.637 

PFCK, TRAN, CFJEX 

.789 

CRW 

PNFC 

.317 

PNFC 

.363 

FEA, CREX, PEX 

.550 

CPF 

PNFC 

.595 

PFCK, PNFC 

.769 

PNFC, CREX, PEX 

.820 

CCAP 

PNFC 

.569 

PGCK, PMFC 

.794 

PI;FC, CREX, PEX 

.852 

CPNF 

PFCK 

.606 

PFCK, PA 

.710 

PFCK, PA, CREX 

.743 

CCOP 

PFCK 

.532 

PFCK, PA 

.624 

. PFCK, TRAN, CREX 

.705 

PLTS 

PFCK 

.597 

PFCK, PNFC 

.741 

PNFC, CREX, PEX 

.796 


i 


«• 


mm 


result In ”pllot error** accusations which may in fact have strong procedural 
compliance implications. 

Schofield recognized that not all of the 97 normal operating procedures 
he identified were equally critical to the safety of flight. To obtain some feel 
for aircrew opinions concerning critlcalityt and concurrently the implied 
importance of compliance, a group of five flight engineers were subsequently 
Invited to rate the criticality of these 97 procedures relative to the safety of 
flight. 

The interesting observations here are: 

1) There is wide disagreement among flight engineers on the 
criticality of most procedures. 

2) Only four of the procedures are unanimously rated at the 
maximum (7) criticality. (Takeoff checklist, takeoff, landing 
checklist and outside scan and visibility callouts. ) 

3) Noise abatement departure procedures received by far the lowest 
ratings. 

4) There are large differences in the scoring tendencies among 
engineers, e.g. some have far more high criticality ratings 
for procedures than others. 
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Vin. CONCLUSIONS 

The project began with an early concern for the dynamics of CIFE’s and 
broad attempts to Identify pertinent research issues. The final products 
were 1) a set of scenarios with associated hardware and techniques for 
studying CIFE phenomena in a simple flight simulator; 2) a set of paper and 
pencil scenarios and associated techniques for studying pilot diagnostic 
strategies and diversion decision making processes; 3) a set of knowledge 
testing instruments designed to measure a pilot*s understanding of aircraft 
subsystems and troubleshooting; 4) a study relating coc ;pit crew procedural 
compliance with performance errors. By-products of this research included 
oneT^I.S. design project, one M.S. thesis, and a Ph.D. dissertation. 

Full Mission Simulation 

Twelve subjects were selected for testing in the full mission GAT scenarios 
Although all were IFR rated, they ranged in age from 20 to 56 years old, in 
flight experience from 270 to 8800 hours and in certification from private 
pilot to ATP. Each subject was given two different forms of the knowledge sur- 
vey to complete and was tiioroughly debriefed after his flight. 

A wide range of cockpit management styles and apparent skill levels were 
observed. Although it was difficult to quantify, ’’good performance” was 
easily recognized by the observers of the experiment. The elements of ’’good 
performance” included: 

1) professional use of the radio 

2) precise heading and altitisde control prior to and during the CIFE 






3) constant awareness of the aircraft position along its Intended route 

4) prompt, but not necessarily Instant, response to the on-set of the 
CIFE (detection) 

5) systematic procedure for trouble shooting 

6) knowledge and use of available ATC resources 

7) diversion decisions which allowed for further potential uncertainties 
The sample was too small to indicate anything other than some initial 

hypotheses concerning pilot performance in such a full-mission setting. How- 
ever, the following tendencies were noted: 

1) Cockpit management style varies widely among pilots. For 
example, some are extremely self-reliant, otliers want 
immediate and extensive help from ATC while still others 
make the decision making process a joint effort with ATC. 

2) Good stick and rudder people seem to have excess capability 
and maintain good stick and rudder performance during and 
after the CIFE. IMore marginal stick and rudder people, on 
the other hand, show increased frequency and amplitude of 
heading and altitude excursions, and experience communications 
difficulties when faced with a CIFE. 

3) Pilots who score well on the knowledge tests teod 

to perform well in problem diagnosis and decision making. 

From the observations of the ex^perimenters and comments made by 
participating subjects, it appears that such a full mission simulation exer^^ise, 


163 


coupled with an appropriate knowledge survey «.nd debriefing, could be a 
valuable tool for rccurr«:nt training of IFR pilots. 

B. P^P Scenarios 

For purposes of analysis the closed-form Icnowledge survey was considered 
to be part of the p/P experiments. This knowledge survey focused on air- 
craft subsystems and trouble shooting in three major areas: 1) engine and 
fvel systems, 2) electrical systems and cockpit Instrumentation, and 3) weather 
and IFR operations. 

A series of Spearman Rank Correlation studies, stepwise regression 
analyses and t-tests were p^Tformed on the combination of pilot background 
variables, knowledge survey results, diagnostic scenario performance and 
decision making measures. Among the observations made from these analyses 
are the following: 

1) There is no correlation between knowledge score and total 
flight hours. 

2) Knowledge score is correlated with pilot ratings held. 

3) Pilots good in one section of the knowledge survey tend to be 
gooa in all sections. 

4) Dlagnosticperformance is highly correlated with knowledge scores. 

5) Knowledge is inversely related to total diagnostic Inqxiiries, 
e.g. , Knowledgeable pilots reach conclusions (right or wrong) 
more rapidly than others. 

G) Total diagnostic Inquiries Is Inversely related to correctness. 
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Tills implies that undirected experimentation Is poor diagnosis 
style. 


7) Total diagnosis correctness score is correlated with efficiency. 

8) Civil trained pilots place a higher worth on ATC service In 
diversion decisions than do military pilots. 

9) Private pilots place a higher worth on weather factors in 
diversion decisions than do commercial and ATP rated pilots. 

10) ATP rated pilots place high worth on time In diversion decisions. 

11) Pilots with good diagnostic scores place less weight on approach 
aids in diversion decisions. 

12) Pilots with good diagnostic scores place more weight on time in 
diversion decisions. 

C. Pro ce dural Compliance 

Schofield used data generated in an experiment conducted In 1976 by 
Dr. H. P. Ruffell Smith to study routine tasks oi flight operations Involving 
airline cockpit crews diring low workload segments of tlmt flight. He was 
concerned with: 

1) Quantifying routine procedures. 

2) Analyzing observed crew errors to ident>'ify which particular crew 
members were tlie primary causes of such errors. 

3) Comparing measures of procedural compliance and operator error. 

The Schofield study of procedural compliance by aircrews who participated 

in the Ruffell Smith experiment suggests the following observations* 
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1) Crew members face an Impossible challenge In attempting to 
mentally catalogue all of the standard operating pnx^edures (SOP) 
published for them, 

2) Routine non-compliance with an assortment of SOP's has been 
documented, 

3) Human redundancy by itself does not erradlcate personnel errors, 
•i) A statistical link appi'ars to exist between operator errors and 

procedural compliance. 
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